2016-11-29 19:22:16

by Benjamin Morgan

[permalink] [raw]
Subject: ath10k firmware crashes in mesh mode on QCA9880

When we try to transmit traffic (ping) between two meshed ath10k devices
running latest lede we keep experiencing ath10k firmware crashes. This
seems to only happen when running in 802.11n/ac mode but not in
802.11a/g mode. Also, from the station dumps it appears that management
traffic is flowing between the devices, however when we try to send
unicast data traffic the firmware crashes immediately.

Platform: Archer C7 AC1750 v2
Software Image: LEDE (HEAD, r2299) Commit:
https://github.com/lede-project/source/commit/d596c21ebd5a3e6ce933eff3e51989031e4b1d58

Crypto: wpa_supplicant
wpa_supplicant-wlan0.conf
network={
ssid="bmorgan_lede_mesh"
key_mgmt=SAE
mode=5
frequency=5180
psk="meshpassword"
}

Backports Verstion:
[ 9.818007] Loading modules backported from Linux version
wt-2016-10-03-1-g6fcb1a6
[ 9.825736] Backport generated by backports.git
backports-20160324-9-g0e38f5c

​​Ath10k Initialization on Station A (dmesg)
[ 9.896715] PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
[ 9.902622] ath10k_pci 0000:01:00.0: pci irq legacy oper_irq_mode 1
irq_mode 0 reset_mode 0
[ 10.123734] ath10k_pci 0000:01:00.0: Direct firmware load for
ath10k/pre-cal-pci-0000:01:00.0.bin failed with error -2
[ 10.134620] ath10k_pci 0000:01:00.0: Falling back to user helper
[ 10.287680] firmware ath10k!pre-cal-pci-0000:01:00.0.bin:
firmware_loading_store: map pages failed
[ 10.622789] ath10k_pci 0000:01:00.0: qca988x hw2.0 target 0x4100016c
chip_id 0x043202ff sub 0000:0000
[ 10.632184] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
tracing 0 dfs 1 testmode 1
[ 10.645231] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api 5
features no-p2p,raw-mode,mfp crc32 9d340dd9
[ 10.655660] ath10k_pci 0000:01:00.0: Direct firmware load for
ath10k/QCA988X/hw2.0/board-2.bin failed with error -2
[ 10.666264] ath10k_pci 0000:01:00.0: Falling back to user helper
[ 10.747925] firmware ath10k!QCA988X!hw2.0!board-2.bin:
firmware_loading_store: map pages failed
[ 11.011123] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
crc32 bebc7c08
[ 12.155224] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
cal file max-sta 128 raw 0 hwcrypto 1

Station A (wlan0):
18:A6:F7:23:6E:66
10.230.5.41

Station B (wlan0):
18:a6:f7:26:0f:21
10.230.5.42

Station Dump on Station A before ping:
Station 18:a6:f7:26:0f:21 (on wlan0)
inactive time:340 ms
rx bytes:2472
rx packets:28
tx bytes:1204
tx packets:9
tx retries:0
tx failed:0
rx drop misc:1
signal: -14 dBm
signal avg:-14 dBm
Toffset:18142530 us
tx bitrate:6.0 MBit/s
rx bitrate:6.0 MBit/s
rx duration:1524 us
mesh llid:0
mesh plid:0
mesh plink:ESTAB
mesh local PS mode:ACTIVE
mesh peer PS mode:UNKNOWN
mesh non-peer PS mode:ACTIVE
authorized:yes
authenticated:yes
associated:yes
preamble:long
WMM/WME:yes
MFP:yes
TDLS peer:no
DTIM period:2
beacon interval:1000
connected time:10 seconds

​Crash Log on Station B (10.230.5.42)
[245.483888] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
2bab5ee9-08ff-4a17-95b1-636d212acebc)
[245.493020] ath10k_pci 0000:01:00.0: qca988x hw2.0 target 0x4100016c
chip_id 0x043202ff sub 0000:0000
[245.502384] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 tracing
0 dfs 1 testmode 1
[245.515436] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api 5
features no-p2p,raw-mode,mfp crc32 9d340dd9
[245.525812] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A crc32
bebc7c08
[245.533232] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2 cal
file max-sta 128 raw 0 hwcrypto 1
[245.544876] ath10k_pci 0000:01:00.0: firmware register dump:
[245.550633] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
0x009A4577 0x00955B31
[245.558676] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
0x00000002 0x00439E98
[245.566715] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
0x00407120 0x004436CC
[245.574749] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
0x009A3518 0x009A3526
[245.582793] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
0x00000000 0x00000000
[245.590836] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
0x0040AC60 0x0040AC09
[245.598882] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
0x00400000 0xC09A4577
[245.606923] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
0x0044110C 0x00442074
[245.614955] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
0x0044110C 0x00407120
[245.623000] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
0x0040AC14 0x00001580
[245.631043] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
0x009C643C 0x004436CC
[245.639086] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
0x004208FC 0x00439E4C
[245.647129] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
0x004208FC 0x00425404
[245.655170] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
0x00424FE8 0x00000002
[245.663198] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
0x00000004 0x004039D0
[245.767023] ieee80211 phy0: Hardware restart was requested
[245.772655] ath10k_pci 0000:01:00.0: failed to synchronize monitor vdev
1 stop: -143
[245.780542] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143

With wpa_supplicant turned off:

Station Dump on Station A before ping:
Station 18:a6:f7:26:0f:21 (on wlan0)
inactive time:60 ms
rx bytes:3874
rx packets:67
tx bytes:707
tx packets:7
tx retries:0
tx failed:1
rx drop misc:3
signal: -14 dBm
signal avg:-14 dBm
Toffset:18446744073684250999 us
tx bitrate:6.0 MBit/s
rx bitrate:6.0 MBit/s
rx duration:1312 us
mesh llid:24953
mesh plid:59412
mesh plink:ESTAB
mesh local PS mode:ACTIVE
mesh peer PS mode:UNKNOWN
mesh non-peer PS mode:ACTIVE
authorized:yes
authenticated:yes
associated:yes
preamble:long
WMM/WME:yes
MFP:no
TDLS peer:no
DTIM period:2
beacon interval:1000
short slot time:yes
connected time:33 seconds

Firmware crash on Station A (10.230.5.41)
[ 1040.984599] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
42d4f6dc-0e89-4505-aefb-58cbb70d0dce)
[ 1040.993732] ath10k_pci 0000:01:00.0: qca988x hw2.0 target 0x4100016c
chip_id 0x043202ff sub 0000:0000
[ 1041.003100] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
tracing 0 dfs 1 testmode 1
[ 1041.016144] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api 5
features no-p2p,raw-mode,mfp crc32 9d340dd9
[ 1041.026523] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
crc32 bebc7c08
[ 1041.033940] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
cal file max-sta 128 raw 0 hwcrypto 1
[ 1041.045593] ath10k_pci 0000:01:00.0: firmware register dump:
[ 1041.051336] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
0x009A4577 0x00955B31
[ 1041.059383] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
0x00000001 0x00435354
[ 1041.067424] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
0x00407120 0x004436CC
[ 1041.075463] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
0x009A43A0 0x009A43DE
[ 1041.083490] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
0x00000000 0x00000000
[ 1041.091535] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
0x0040AC60 0x0040AC09
[ 1041.099579] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
0x00955A00 0xC09A4577
[ 1041.107622] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
0x0044110C 0x00442074
[ 1041.115661] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
0x0044110C 0x00407120
[ 1041.123689] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
0x0040AC10 0x00001580
[ 1041.131733] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
0x009C643C 0x004436CC
[ 1041.139777] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
0x004208FC 0x00439E4C
[ 1041.147820] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
0x004208FC 0x00425758
[ 1041.155860] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
0x00424FE8 0x00000002
[ 1041.163888] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
0x00000004 0x004039D0
[ 1041.267025] ieee80211 phy0: Hardware restart was requested
[ 1041.272656] ath10k_pci 0000:01:00.0: failed to synchronize monitor
vdev 1 stop: -143
[ 1041.280545] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143


Thanks in advance for your time and help,

~Benjamin


2016-12-13 18:42:30

by Benjamin Morgan

[permalink] [raw]
Subject: Re: ath10k firmware crashes in mesh mode on QCA9880

Just tested the latest 10.2.4.70.59-2 firmware and it still crashes with
wpa_supplicant encrypted mesh =(

[ 85.201440] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
b7f44483-0488-46af-8dff-db88f4b56327)
[ 85.210573] ath10k_pci 0000:01:00.0: qca988x hw2.0 target 0x4100016c
chip_id 0x043202ff sub 0000:0000
[ 85.219940] ath10k_pci 0000:01:00.0: kconfig debug 1 debugfs 1
tracing 0 dfs 1 testmode 1
[ 85.233034] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.59-2 api
5 features no-p2p,raw-mode,mfp,allows-mesh-bcast crc32 4159f498
[ 85.245177] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
crc32 bebc7c08
[ 85.252592] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
cal file max-sta 128 raw 0 hwcrypto 1
[ 85.264235] ath10k_pci 0000:01:00.0: firmware register dump:
[ 85.269992] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
0x009A45AF 0x00955B31
[ 85.278031] ath10k_pci 0000:01:00.0: [04]: 0x009A45AF 0x00060130
0x00000002 0x00439E98
[ 85.286078] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
0x00407120 0x004436CC
[ 85.294107] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
0x009A3550 0x009A355E
[ 85.302152] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
0x00000000 0x00000000
[ 85.310195] ath10k_pci 0000:01:00.0: [20]: 0x409A45AF 0x0040AAC4
0x0040AC60 0x0040AC09
[ 85.318239] ath10k_pci 0000:01:00.0: [24]: 0x809A44F2 0x0040AB24
0x00400000 0xC09A45AF
[ 85.326282] ath10k_pci 0000:01:00.0: [28]: 0x809A3A16 0x0040AB84
0x0044110C 0x00442074
[ 85.334314] ath10k_pci 0000:01:00.0: [32]: 0x809A601A 0x0040ABB4
0x0044110C 0x00407120
[ 85.342350] ath10k_pci 0000:01:00.0: [36]: 0x809A2EA4 0x0040ABF4
0x0040AC14 0x00001580
[ 85.350393] ath10k_pci 0000:01:00.0: [40]: 0x80990F63 0x0040AD04
0x009C6458 0x004436CC
[ 85.358437] ath10k_pci 0000:01:00.0: [44]: 0x80998520 0x0040AD64
0x004208FC 0x00439E4C
[ 85.366479] ath10k_pci 0000:01:00.0: [48]: 0x8099AEA5 0x0040AD84
0x004208FC 0x00425AAC
[ 85.374512] ath10k_pci 0000:01:00.0: [52]: 0x809BFC39 0x0040AEE4
0x00424FE8 0x00000002
[ 85.382548] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
0x00000004 0x004039D0
[ 85.487067] ieee80211 phy0: Hardware restart was requested
[ 85.492701] ath10k_pci 0000:01:00.0: wmi disable pktlog

Any new leads on tracking down this issue?

~Benjamin


On 12/06/2016 01:32 PM, Benjamin Morgan wrote:
> 1. Yes, this appears to happens every time a unicast packet with
> wpa_supplicant encryption in VHT80 mode is received. I haven't seen a
> successful ping-pong pair.
> 2. We tried with 10.2.4.70.42-2 firmware and still saw crashes.
> 3. We ran our experiment again with extra debugging turned on.
> Node A: 18:A6:F7:23:6E:66 | 10.230.5.41
> Node B: 18:A6:F7:26:0F:21 | 10.230.5.42
> The ping command we used was run on Node A was 'ping -s 1500 -i
> 0.1 10.230.5.42'
> Here is the dmesg log from Node B.
>
> [ 5413.478170] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5413.503954] ath10k_pci 0000:01:00.0: scan event bss channel type 4
> reason 3 freq 5825 req_id 40961 scan_id 40960 vdev_id 0 state running (2)
> [ 5413.503985] ath10k_pci 0000:01:00.0: chan info err_code 0 freq 5825
> cmd_flags 1 noise_floor -105 rx_clear_count 7692807 cycle_count 312271423
> [ 5413.504029] ath10k_pci 0000:01:00.0: scan event completed type 2
> reason 0 freq 5825 req_id 40961 scan_id 40960 vdev_id 0 state running (2)
> [ 5413.525868] ath10k_pci 0000:01:00.0: wmi vdev install key idx 1
> cipher 4 len 16
> [ 5413.526014] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 31
> value 1
> [ 5413.526193] ath10k_pci 0000:01:00.0: mac vdev 0 set keyidx 1
> [ 5413.526216] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 31
> value 1
> [ 5413.526532] ath10k_pci 0000:01:00.0: mac chanctx add freq 5180
> width 3 ptr 86db29b0
> [ 5413.526556] ath10k_pci 0000:01:00.0: mac monitor recalc started? 0
> needed? 0 allowed? 1
> [ 5413.526574] ath10k_pci 0000:01:00.0: mac chanctx assign ptr
> 86db29b0 vdev_id 0
> [ 5413.526592] ath10k_pci 0000:01:00.0: mac vdev 0 start center_freq
> 5180 phymode 11ac-vht80
> [ 5413.526616] ath10k_pci 0000:01:00.0: wmi vdev start id 0x0 flags:
> 0x0, freq 5180, mode 10, ch_flags: 0xA000000, max_power: 46
> [ 5413.533099] ath10k_pci 0000:01:00.0: WMI_VDEV_START_RESP_EVENTID
> [ 5413.533148] ath10k_pci 0000:01:00.0: mac vdev_id 0 txpower 23
> [ 5413.533163] ath10k_pci 0000:01:00.0: mac txpower 23
> [ 5413.533180] ath10k_pci 0000:01:00.0: wmi pdev set param 3 value 46
> [ 5413.533247] ath10k_pci 0000:01:00.0: wmi pdev set param 4 value 46
> [ 5413.533295] ath10k_pci 0000:01:00.0: mac chanctx change freq 5180
> width 3 ptr 86db29b0 changed 10
> [ 5413.533318] ath10k_pci 0000:01:00.0: mac chanctx change freq 5180
> width 3 ptr 86db29b0 changed 2
> [ 5413.533337] ath10k_pci 0000:01:00.0: mac monitor recalc started? 0
> needed? 1 allowed? 1
> [ 5413.533357] ath10k_pci 0000:01:00.0: WMI vdev create: id 1 type 4
> subtype 0 macaddr 18:a6:f7:26:0f:21
> [ 5413.533412] ath10k_pci 0000:01:00.0: mac monitor vdev 1 created
> [ 5413.533463] ath10k_pci 0000:01:00.0: wmi vdev start id 0x1 flags:
> 0x0, freq 5180, mode 10, ch_flags: 0xA000000, max_power: 46
> [ 5413.937652] ath10k_pci 0000:01:00.0: wmi event debug mesg len 152
> [ 5413.978273] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5414.478363] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5414.527015] ath10k_pci 0000:01:00.0: WMI_VDEV_START_RESP_EVENTID
> [ 5414.527067] ath10k_pci 0000:01:00.0: wmi mgmt vdev up id 0x1 assoc
> id 0 bssid 18:a6:f7:26:0f:21
> [ 5414.527121] ath10k_pci 0000:01:00.0: mac monitor vdev 1 started
> [ 5414.527165] ath10k_pci 0000:01:00.0: mac monitor started
> [ 5414.527216] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 3
> value 1000
> [ 5414.527262] ath10k_pci 0000:01:00.0: mac vdev 0 beacon_interval 1000
> [ 5414.527278] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
> staggered
> [ 5414.527294] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
> [ 5414.527314] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
> [ 5414.527330] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
> value 2
> [ 5414.527457] ath10k_pci 0000:01:00.0: wmi mgmt vdev up id 0x0 assoc
> id 0 bssid 00:00:00:00:00:00
> [ 5414.527501] ath10k_pci 0000:01:00.0: mac vdev 0 up
> [ 5414.527564] ath10k_pci 0000:01:00.0: WMI_TBTTOFFSET_UPDATE_EVENTID
> [ 5414.541090] ath10k_pci 0000:01:00.0: mac monitor recalc started? 1
> needed? 1 allowed? 1
> [ 5414.978454] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5415.478548] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5415.978649] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5416.445280] ath10k_pci 0000:01:00.0: mac monitor recalc started? 1
> needed? 1 allowed? 1
> [ 5416.478761] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5416.978879] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5417.478985] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5417.979081] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5418.479190] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5418.979301] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5419.479403] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5419.979551] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5420.479643] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5420.979746] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5421.479841] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5421.979940] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5422.480288] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5422.980386] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5423.480490] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5423.980600] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5424.480702] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5424.971969] ath10k_pci 0000:01:00.0: mac vdev 0 peer create
> 18:a6:f7:23:6e:66 (new sta) sta 1 / 128 peer 2 / 144
> [ 5424.972000] ath10k_pci 0000:01:00.0: wmi peer create vdev_id 0
> peer_addr 18:a6:f7:23:6e:66
> [ 5424.975107] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
> staggered
> [ 5424.975134] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
> [ 5424.975219] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
> [ 5424.975238] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
> value 2
> [ 5424.980787] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5425.204468] ath10k_pci 0000:01:00.0: mac sta 18:a6:f7:23:6e:66
> associated
> [ 5425.204531] ath10k_pci 0000:01:00.0: mac ht peer 18:a6:f7:23:6e:66
> mcs cnt 24 nss 3
> [ 5425.204548] ath10k_pci 0000:01:00.0: mac peer 18:a6:f7:23:6e:66 qos 1
> [ 5425.204563] ath10k_pci 0000:01:00.0: mac peer 18:a6:f7:23:6e:66
> phymode 11na-ht40
> [ 5425.204585] ath10k_pci 0000:01:00.0: wmi peer assoc vdev 0 addr
> 18:a6:f7:23:6e:66 (new)
> [ 5425.204614] ath10k_pci 0000:01:00.0: wmi vdev 0 peer
> 0x18:a6:f7:23:6e:66 set param 1 value 0
> [ 5425.205376] ath10k_pci 0000:01:00.0: received event id 36891 not
> implemented
> [ 5425.209240] ath10k_pci 0000:01:00.0: wmi vdev install key idx 0
> cipher 4 len 16
> [ 5425.209655] ath10k_pci 0000:01:00.0: wmi vdev install key idx 1
> cipher 4 len 16
> [ 5425.209848] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 31
> value 1
> [ 5425.210196] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
> staggered
> [ 5425.210221] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
> [ 5425.210296] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
> [ 5425.210315] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
> value 2
> [ 5425.480863] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5425.938619] ath10k_pci 0000:01:00.0: wmi event debug mesg len 100
> [ 5425.980946] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5425.995007] ath10k_pci 0000:01:00.0: mac sta rc update for
> 18:a6:f7:23:6e:66 changed 00000001 bw 2 nss 3 smps 1
> [ 5425.995060] ath10k_pci 0000:01:00.0: mac update sta
> 18:a6:f7:23:6e:66 peer bw 2
> [ 5425.995081] ath10k_pci 0000:01:00.0: wmi vdev 0 peer
> 0x18:a6:f7:23:6e:66 set param 4 value 2
> [ 5426.481030] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5426.981117] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5427.481206] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5427.981294] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5428.481628] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5428.981718] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5429.481812] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5429.981894] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5430.481985] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5430.982073] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5431.482174] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5431.982505] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5432.482597] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5432.982679] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5433.482765] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5433.982857] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5434.482946] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5434.983008] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5435.483100] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5435.983181] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5436.483276] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5436.983366] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5437.483445] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5437.983516] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5438.483607] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5438.983692] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 5439.439875] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
> db76b67c-ca98-4519-a762-4ff4edb45526)
> [ 5439.449007] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> 0x4100016c chip_id 0x043202ff sub 0000:0000
> [ 5439.458378] ath10k_pci 0000:01:00.0: kconfig debug 1 debugfs 1
> tracing 0 dfs 1 testmode 1
> [ 5439.471460] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api
> 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
> [ 5439.481844] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> crc32 bebc7c08
> [ 5439.489267] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
> cal file max-sta 128 raw 0 hwcrypto 1
> [ 5439.500918] ath10k_pci 0000:01:00.0: firmware register dump:
> [ 5439.506678] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
> 0x009A4577 0x00955B31
> [ 5439.514706] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
> 0x00000002 0x00439E98
> [ 5439.522751] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
> 0x00407120 0x004436CC
> [ 5439.530794] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
> 0x009A3518 0x009A3526
> [ 5439.538834] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
> 0x00000000 0x00000000
> [ 5439.546871] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
> 0x0040AC60 0x0040AC09
> [ 5439.554915] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
> 0x00400000 0xC09A4577
> [ 5439.562948] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
> 0x0044110C 0x00442074
> [ 5439.570992] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
> 0x0044110C 0x00407120
> [ 5439.579032] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
> 0x0040AC14 0x00001580
> [ 5439.587070] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
> 0x009C643C 0x004436CC
> [ 5439.595113] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
> 0x004208FC 0x00439E4C
> [ 5439.603146] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
> 0x004208FC 0x00425E00
> [ 5439.611191] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
> 0x00424FE8 0x00000002
> [ 5439.619230] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
> 0x00000004 0x004039D0
> [ 5439.726818] ieee80211 phy0: Hardware restart was requested
> [ 5439.732433] ath10k_pci 0000:01:00.0: wmi mgmt vdev down id 0x1
> [ 5439.732461] ath10k_pci 0000:01:00.0: wmi vdev stop id 0x1
> [ 5439.732482] ath10k_pci 0000:01:00.0: failed to synchronize monitor
> vdev 1 stop: -143
> [ 5439.740370] ath10k_pci 0000:01:00.0: mac monitor vdev 1 stopped
> [ 5439.740386] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
> [ 5439.747042] ath10k_pci 0000:01:00.0: wmi disable pktlog
>
> We noticed in this log that when the radio starts up it says that it
> is in VHT80 mode:
> [ 5413.526592] ath10k_pci 0000:01:00.0: mac vdev 0 start center_freq
> 5180 phymode 11ac-vht80
>
> But when a peer connects it seems to think the peer is in HT40 mode:
> [ 5425.204563] ath10k_pci 0000:01:00.0: mac peer 18:a6:f7:23:6e:66
> phymode 11na-ht40
>
> Compared to no encryption case - this log was taken from Node A:
>
> [ 24.874253] ath10k_pci 0000:01:00.0: mac chanctx add freq 5180
> width 3 ptr 86d26db0
> [ 24.874278] ath10k_pci 0000:01:00.0: mac monitor recalc started? 0
> needed? 0 allowed? 1
> [ 24.874296] ath10k_pci 0000:01:00.0: mac chanctx assign ptr
> 86d26db0 vdev_id 0
> [ 24.874312] ath10k_pci 0000:01:00.0: mac vdev 0 start center_freq
> 5180 phymode 11ac-vht80
> [ 24.874337] ath10k_pci 0000:01:00.0: wmi vdev start id 0x0 flags:
> 0x0, freq 5180, mode 10, ch_flags: 0xA000000, max_power: 46
> [ 24.881335] ath10k_pci 0000:01:00.0: WMI_VDEV_START_RESP_EVENTID
> [ 24.881423] ath10k_pci 0000:01:00.0: mac vdev_id 0 txpower 23
> [ 24.881438] ath10k_pci 0000:01:00.0: mac txpower 23
> [ 24.881454] ath10k_pci 0000:01:00.0: wmi pdev set param 3 value 46
> [ 24.881491] ath10k_pci 0000:01:00.0: wmi pdev set param 4 value 46
> [ 24.881515] ath10k_pci 0000:01:00.0: mac chanctx change freq 5180
> width 3 ptr 86d26db0 changed 10
> [ 24.881535] ath10k_pci 0000:01:00.0: mac chanctx change freq 5180
> width 3 ptr 86d26db0 changed 2
> [ 24.881554] ath10k_pci 0000:01:00.0: mac monitor recalc started? 0
> needed? 1 allowed? 1
> [ 24.881574] ath10k_pci 0000:01:00.0: WMI vdev create: id 1 type 4
> subtype 0 macaddr 18:a6:f7:23:6e:66
> [ 24.881689] ath10k_pci 0000:01:00.0: mac monitor vdev 1 created
> [ 24.881745] ath10k_pci 0000:01:00.0: wmi vdev start id 0x1 flags:
> 0x0, freq 5180, mode 10, ch_flags: 0xA000000, max_power: 46
> [ 25.273460] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 25.730570] ath10k_pci 0000:01:00.0: wmi event debug mesg len 300
> [ 25.773566] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 25.874556] ath10k_pci 0000:01:00.0: WMI_VDEV_START_RESP_EVENTID
> [ 25.879992] ath10k_pci 0000:01:00.0: wmi mgmt vdev up id 0x1 assoc
> id 0 bssid 18:a6:f7:23:6e:66
> [ 25.880077] ath10k_pci 0000:01:00.0: mac monitor vdev 1 started
> [ 25.880093] ath10k_pci 0000:01:00.0: mac monitor started
> [ 25.880139] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 3
> value 1000
> [ 25.880184] ath10k_pci 0000:01:00.0: mac vdev 0 beacon_interval 1000
> [ 25.880199] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
> staggered
> [ 25.880215] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
> [ 25.880235] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
> [ 25.880250] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
> value 2
> [ 25.880988] ath10k_pci 0000:01:00.0: wmi mgmt vdev up id 0x0 assoc
> id 0 bssid 00:00:00:00:00:00
> [ 25.881035] ath10k_pci 0000:01:00.0: mac vdev 0 up
> [ 25.881097] ath10k_pci 0000:01:00.0: WMI_TBTTOFFSET_UPDATE_EVENTID
> [ 25.882968] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
> [ 25.928796] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
> staggered
> [ 25.928821] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
> [ 25.928866] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
> [ 25.928883] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
> value 2
> [ 25.929020] ath10k_pci 0000:01:00.0: mac monitor recalc started? 1
> needed? 1 allowed? 1
> [ 25.941886] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
> staggered
> [ 25.941911] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
> [ 25.941955] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
> [ 25.941972] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
> value 2
> [ 25.953727] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
> staggered
> [ 25.953753] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
> [ 25.953798] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
> [ 25.953817] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
> value 2
> [ 25.970588] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
> staggered
> [ 25.970614] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
> [ 25.970659] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
> [ 25.970676] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
> value 2
> [ 25.989056] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
> staggered
> [ 25.989081] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
> [ 25.989126] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
> [ 25.989143] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
> value 2
> [ 26.071686] ath10k_pci 0000:01:00.0: mac vdev 0 peer create
> 18:a6:f7:26:0f:21 (new sta) sta 1 / 128 peer 2 / 144
> [ 26.071712] ath10k_pci 0000:01:00.0: wmi peer create vdev_id 0
> peer_addr 18:a6:f7:26:0f:21
> [ 26.071952] ath10k_pci 0000:01:00.0: mac sta 18:a6:f7:26:0f:21
> associated
> [ 26.071981] ath10k_pci 0000:01:00.0: mac ht peer 18:a6:f7:26:0f:21
> mcs cnt 24 nss 3
> [ 26.071999] ath10k_pci 0000:01:00.0: mac vht peer 18:a6:f7:26:0f:21
> max_mpdu 1048575 flags 0x601b001
> [ 26.072013] ath10k_pci 0000:01:00.0: mac peer 18:a6:f7:26:0f:21 qos 1
> [ 26.072028] ath10k_pci 0000:01:00.0: mac peer 18:a6:f7:26:0f:21
> phymode 11ac-vht80
> [ 26.072047] ath10k_pci 0000:01:00.0: wmi peer assoc vdev 0 addr
> 18:a6:f7:26:0f:21 (new)
> [ 26.072071] ath10k_pci 0000:01:00.0: wmi vdev 0 peer
> 0x18:a6:f7:26:0f:21 set param 1 value 0
> [ 26.072502] ath10k_pci 0000:01:00.0: received event id 36891 not
> implemented
> [ 26.074194] ath10k_pci 0000:01:00.0: mac sta rc update for
> 18:a6:f7:26:0f:21 changed 00000000 bw 2 nss 3 smps 1
> [ 26.074586] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
> staggered
> [ 26.074609] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
> [ 26.074682] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
> [ 26.074701] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
> value 2
> [ 26.074760] ath10k_pci 0000:01:00.0: mac vdev 0 slot_time 2
> [ 26.074779] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 7
> value 2
> [ 26.273652] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 26.730650] ath10k_pci 0000:01:00.0: wmi event debug mesg len 44
> [ 26.773733] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
> [ 27.135445] ath10k_pci 0000:01:00.0: mac monitor recalc started? 1
> needed? 1 allowed? 1
> [ 27.273810] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>
> It seems to start up in VHT80 mode and when it peers with Node B it
> thinks Node B is also in VHT80 mode and ping works.
>
> 4. Beacons are sent at 6 Mb/s basic rate and unicast QoS Data is sent
> with three spatial streams. Attached is the full pcap of the experiment.
>
> Thank you for looking into this!
>
> ~Benjamin
>
> On 12/05/2016 11:24 AM, Nagarajan, Ashok Raj wrote:
>> 0x009A4577 0x00955B31
>>
>> Benjamin, Thanks for the logs.
>> Quick questions to further debug the issue here,
>>
>> 1. Is this issue seen every time you start sending data traffic?
>> 2. Issue seen with older firmwares? (FYR,
>> http://linuxwireless.org/en/users/Drivers/ath10k/firmware/ )
>> 3. Could you please share the dmesg from your device after enabling
>> MAC and WMI logs in ath10k driver
>> To enable debug logs please see
>> http://linuxwireless.org/en/users/Drivers/ath10k/debug/
>> 4. Do you know what is the Number of Spatial Streams seen in mesh
>> beacons and in mesh data packet?
>>
>> Thanks,
>> Ashok
>

2016-12-03 01:28:05

by Benjamin Morgan

[permalink] [raw]
Subject: Re: ath10k firmware crashes in mesh mode on QCA9880

Just tried 10.2.4.70.58 firmware that you linked to and it still crashes:

[ 131.568989] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
1838347e-9380-4a26-ac9d-2963ee95968b)
[ 131.578124] ath10k_pci 0000:01:00.0: qca988x hw2.0 target 0x4100016c
chip_id 0x043202ff sub 0000:0000
[ 131.587491] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
tracing 0 dfs 1 testmode 1
[ 131.600521] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.58 api 5
features no-p2p,raw-mode,mfp crc32 e1af076f
[ 131.610899] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
crc32 bebc7c08
[ 131.618325] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
cal file max-sta 128 raw 0 hwcrypto 1
[ 131.629965] ath10k_pci 0000:01:00.0: firmware register dump:
[ 131.635728] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
0x009A45AF 0x00955B31
[ 131.643761] ath10k_pci 0000:01:00.0: [04]: 0x009A45AF 0x00060130
0x00000002 0x00439E98
[ 131.651806] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
0x00407120 0x004436CC
[ 131.659852] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
0x009A3550 0x009A355E
[ 131.667892] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x009A31D6
0x00000000 0x00000000
[ 131.675936] ath10k_pci 0000:01:00.0: [20]: 0x409A45AF 0x0040AAC4
0x0040AC60 0x0040AC09
[ 131.683968] ath10k_pci 0000:01:00.0: [24]: 0x809A44F2 0x0040AB24
0x00400000 0xC09A45AF
[ 131.692013] ath10k_pci 0000:01:00.0: [28]: 0x809A3A16 0x0040AB84
0x0044110C 0x00442074
[ 131.700056] ath10k_pci 0000:01:00.0: [32]: 0x809A601A 0x0040ABB4
0x0044110C 0x00407120
[ 131.708100] ath10k_pci 0000:01:00.0: [36]: 0x809A2EA4 0x0040ABF4
0x0040AC14 0x00001580
[ 131.716143] ath10k_pci 0000:01:00.0: [40]: 0x80990F63 0x0040AD04
0x009C6458 0x004436CC
[ 131.724175] ath10k_pci 0000:01:00.0: [44]: 0x80998520 0x0040AD64
0x004208FC 0x00439E4C
[ 131.732220] ath10k_pci 0000:01:00.0: [48]: 0x8099AEA5 0x0040AD84
0x004208FC 0x00425874
[ 131.740263] ath10k_pci 0000:01:00.0: [52]: 0x809BFC39 0x0040AEE4
0x00424FE8 0x00000002
[ 131.748306] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
0x00000004 0x004039D0
[ 131.857076] ieee80211 phy0: Hardware restart was requested
[ 131.862705] ath10k_pci 0000:01:00.0: failed to synchronize monitor
vdev 1 stop: -143
[ 131.870594] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143

~Benjamin

On 11/30/2016 11:09 PM, Mohammed Shafi Shajakhan wrote:
> Hi,
>
> On Tue, Nov 29, 2016 at 11:22:12AM -0800, Benjamin Morgan wrote:
>> When we try to transmit traffic (ping) between two meshed ath10k
>> devices running latest lede we keep experiencing ath10k firmware
>> crashes. This seems to only happen when running in 802.11n/ac mode
>> but not in 802.11a/g mode. Also, from the station dumps it appears
>> that management traffic is flowing between the devices, however when
>> we try to send unicast data traffic the firmware crashes
>> immediately.
> [shafi] Did you get a chance to try with the below firmware as well
> https://github.com/kvalo/ath10k-firmware/blob/master/QCA988X/hw2.0/10.2.4.70/firmware-5.bin_10.2.4.70.58
>
>> Platform: Archer C7 AC1750 v2
>> Software Image: LEDE (HEAD, r2299) Commit: https://github.com/lede-project/source/commit/d596c21ebd5a3e6ce933eff3e51989031e4b1d58
>>
>> Crypto: wpa_supplicant
>> wpa_supplicant-wlan0.conf
>> network={
>> ssid="bmorgan_lede_mesh"
>> key_mgmt=SAE
>> mode=5
>> frequency=5180
>> psk="meshpassword"
>> }
>>
>> Backports Verstion:
>> [ 9.818007] Loading modules backported from Linux version
>> wt-2016-10-03-1-g6fcb1a6
>> [ 9.825736] Backport generated by backports.git
>> backports-20160324-9-g0e38f5c
>>
>> ​​Ath10k Initialization on Station A (dmesg)
>> [ 9.896715] PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
>> [ 9.902622] ath10k_pci 0000:01:00.0: pci irq legacy oper_irq_mode
>> 1 irq_mode 0 reset_mode 0
>> [ 10.123734] ath10k_pci 0000:01:00.0: Direct firmware load for
>> ath10k/pre-cal-pci-0000:01:00.0.bin failed with error -2
>> [ 10.134620] ath10k_pci 0000:01:00.0: Falling back to user helper
>> [ 10.287680] firmware ath10k!pre-cal-pci-0000:01:00.0.bin:
>> firmware_loading_store: map pages failed
>> [ 10.622789] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
>> 0x4100016c chip_id 0x043202ff sub 0000:0000
>> [ 10.632184] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
>> tracing 0 dfs 1 testmode 1
>> [ 10.645231] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
>> api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
>> [ 10.655660] ath10k_pci 0000:01:00.0: Direct firmware load for
>> ath10k/QCA988X/hw2.0/board-2.bin failed with error -2
>> [ 10.666264] ath10k_pci 0000:01:00.0: Falling back to user helper
>> [ 10.747925] firmware ath10k!QCA988X!hw2.0!board-2.bin:
>> firmware_loading_store: map pages failed
>> [ 11.011123] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
>> crc32 bebc7c08
>> [ 12.155224] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
>> 2 cal file max-sta 128 raw 0 hwcrypto 1
>>
>> Station A (wlan0):
>> 18:A6:F7:23:6E:66
>> 10.230.5.41
>>
>> Station B (wlan0):
>> 18:a6:f7:26:0f:21
>> 10.230.5.42
>>
>> Station Dump on Station A before ping:
>> Station 18:a6:f7:26:0f:21 (on wlan0)
>> inactive time:340 ms
>> rx bytes:2472
>> rx packets:28
>> tx bytes:1204
>> tx packets:9
>> tx retries:0
>> tx failed:0
>> rx drop misc:1
>> signal: -14 dBm
>> signal avg:-14 dBm
>> Toffset:18142530 us
>> tx bitrate:6.0 MBit/s
>> rx bitrate:6.0 MBit/s
>> rx duration:1524 us
>> mesh llid:0
>> mesh plid:0
>> mesh plink:ESTAB
>> mesh local PS mode:ACTIVE
>> mesh peer PS mode:UNKNOWN
>> mesh non-peer PS mode:ACTIVE
>> authorized:yes
>> authenticated:yes
>> associated:yes
>> preamble:long
>> WMM/WME:yes
>> MFP:yes
>> TDLS peer:no
>> DTIM period:2
>> beacon interval:1000
>> connected time:10 seconds
>>
>> ​Crash Log on Station B (10.230.5.42)
>> [245.483888] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
>> 2bab5ee9-08ff-4a17-95b1-636d212acebc)
>> [245.493020] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
>> 0x4100016c chip_id 0x043202ff sub 0000:0000
>> [245.502384] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
>> tracing 0 dfs 1 testmode 1
>> [245.515436] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api
>> 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
>> [245.525812] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
>> crc32 bebc7c08
>> [245.533232] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
>> cal file max-sta 128 raw 0 hwcrypto 1
>> [245.544876] ath10k_pci 0000:01:00.0: firmware register dump:
>> [245.550633] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
>> 0x009A4577 0x00955B31
>> [245.558676] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
>> 0x00000002 0x00439E98
>> [245.566715] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
>> 0x00407120 0x004436CC
>> [245.574749] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
>> 0x009A3518 0x009A3526
>> [245.582793] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
>> 0x00000000 0x00000000
>> [245.590836] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
>> 0x0040AC60 0x0040AC09
>> [245.598882] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
>> 0x00400000 0xC09A4577
>> [245.606923] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
>> 0x0044110C 0x00442074
>> [245.614955] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
>> 0x0044110C 0x00407120
>> [245.623000] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
>> 0x0040AC14 0x00001580
>> [245.631043] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
>> 0x009C643C 0x004436CC
>> [245.639086] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
>> 0x004208FC 0x00439E4C
>> [245.647129] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
>> 0x004208FC 0x00425404
>> [245.655170] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
>> 0x00424FE8 0x00000002
>> [245.663198] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
>> 0x00000004 0x004039D0
>> [245.767023] ieee80211 phy0: Hardware restart was requested
>> [245.772655] ath10k_pci 0000:01:00.0: failed to synchronize monitor
>> vdev 1 stop: -143
>> [245.780542] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
>> ​
>> With wpa_supplicant turned off:
>>
>> Station Dump on Station A before ping:
>> Station 18:a6:f7:26:0f:21 (on wlan0)
>> inactive time:60 ms
>> rx bytes:3874
>> rx packets:67
>> tx bytes:707
>> tx packets:7
>> tx retries:0
>> tx failed:1
>> rx drop misc:3
>> signal: -14 dBm
>> signal avg:-14 dBm
>> Toffset:18446744073684250999 us
>> tx bitrate:6.0 MBit/s
>> rx bitrate:6.0 MBit/s
>> rx duration:1312 us
>> mesh llid:24953
>> mesh plid:59412
>> mesh plink:ESTAB
>> mesh local PS mode:ACTIVE
>> mesh peer PS mode:UNKNOWN
>> mesh non-peer PS mode:ACTIVE
>> authorized:yes
>> authenticated:yes
>> associated:yes
>> preamble:long
>> WMM/WME:yes
>> MFP:no
>> TDLS peer:no
>> DTIM period:2
>> beacon interval:1000
>> short slot time:yes
>> connected time:33 seconds
>>
>> Firmware crash on Station A (10.230.5.41)
>> [ 1040.984599] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
>> 42d4f6dc-0e89-4505-aefb-58cbb70d0dce)
>> [ 1040.993732] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
>> 0x4100016c chip_id 0x043202ff sub 0000:0000
>> [ 1041.003100] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
>> tracing 0 dfs 1 testmode 1
>> [ 1041.016144] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
>> api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
>> [ 1041.026523] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
>> crc32 bebc7c08
>> [ 1041.033940] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
>> 2 cal file max-sta 128 raw 0 hwcrypto 1
>> [ 1041.045593] ath10k_pci 0000:01:00.0: firmware register dump:
>> [ 1041.051336] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
>> 0x009A4577 0x00955B31
>> [ 1041.059383] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
>> 0x00000001 0x00435354
>> [ 1041.067424] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
>> 0x00407120 0x004436CC
>> [ 1041.075463] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
>> 0x009A43A0 0x009A43DE
>> [ 1041.083490] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
>> 0x00000000 0x00000000
>> [ 1041.091535] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
>> 0x0040AC60 0x0040AC09
>> [ 1041.099579] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
>> 0x00955A00 0xC09A4577
>> [ 1041.107622] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
>> 0x0044110C 0x00442074
>> [ 1041.115661] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
>> 0x0044110C 0x00407120
>> [ 1041.123689] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
>> 0x0040AC10 0x00001580
>> [ 1041.131733] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
>> 0x009C643C 0x004436CC
>> [ 1041.139777] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
>> 0x004208FC 0x00439E4C
>> [ 1041.147820] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
>> 0x004208FC 0x00425758
>> [ 1041.155860] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
>> 0x00424FE8 0x00000002
>> [ 1041.163888] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
>> 0x00000004 0x004039D0
>> [ 1041.267025] ieee80211 phy0: Hardware restart was requested
>> [ 1041.272656] ath10k_pci 0000:01:00.0: failed to synchronize
>> monitor vdev 1 stop: -143
>> [ 1041.280545] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
>>
>> ​
>> Thanks in advance for your time and help,
>>
>> ~Benjamin
>>

2016-12-03 12:46:12

by Mohammed Shafi Shajakhan

[permalink] [raw]
Subject: Re: ath10k firmware crashes in mesh mode on QCA9880

https://patchwork.kernel.org/patch/9437519/
(sorry missed this in the previous thread)

On Sat, Dec 03, 2016 at 06:13:58PM +0530, Mohammed Shafi Shajakhan wrote:
> Hi Benjamin,
>
> On Fri, Dec 02, 2016 at 05:28:02PM -0800, Benjamin Morgan wrote:
> > Just tried 10.2.4.70.58 firmware that you linked to and it still crashes:
> >
> > [ 131.568989] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
> > 1838347e-9380-4a26-ac9d-2963ee95968b)
> > [ 131.578124] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> > 0x4100016c chip_id 0x043202ff sub 0000:0000
> > [ 131.587491] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
> > tracing 0 dfs 1 testmode 1
> > [ 131.600521] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.58
> > api 5 features no-p2p,raw-mode,mfp crc32 e1af076f
> > [ 131.610899] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> > crc32 bebc7c08
> > [ 131.618325] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
> > 2 cal file max-sta 128 raw 0 hwcrypto 1
> > [ 131.629965] ath10k_pci 0000:01:00.0: firmware register dump:
> > [ 131.635728] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
> > 0x009A45AF 0x00955B31
> > [ 131.643761] ath10k_pci 0000:01:00.0: [04]: 0x009A45AF 0x00060130
> > 0x00000002 0x00439E98
> > [ 131.651806] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
> > 0x00407120 0x004436CC
> > [ 131.659852] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
> > 0x009A3550 0x009A355E
> > [ 131.667892] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x009A31D6
> > 0x00000000 0x00000000
> > [ 131.675936] ath10k_pci 0000:01:00.0: [20]: 0x409A45AF 0x0040AAC4
> > 0x0040AC60 0x0040AC09
> > [ 131.683968] ath10k_pci 0000:01:00.0: [24]: 0x809A44F2 0x0040AB24
> > 0x00400000 0xC09A45AF
> > [ 131.692013] ath10k_pci 0000:01:00.0: [28]: 0x809A3A16 0x0040AB84
> > 0x0044110C 0x00442074
> > [ 131.700056] ath10k_pci 0000:01:00.0: [32]: 0x809A601A 0x0040ABB4
> > 0x0044110C 0x00407120
> > [ 131.708100] ath10k_pci 0000:01:00.0: [36]: 0x809A2EA4 0x0040ABF4
> > 0x0040AC14 0x00001580
> > [ 131.716143] ath10k_pci 0000:01:00.0: [40]: 0x80990F63 0x0040AD04
> > 0x009C6458 0x004436CC
> > [ 131.724175] ath10k_pci 0000:01:00.0: [44]: 0x80998520 0x0040AD64
> > 0x004208FC 0x00439E4C
> > [ 131.732220] ath10k_pci 0000:01:00.0: [48]: 0x8099AEA5 0x0040AD84
> > 0x004208FC 0x00425874
> > [ 131.740263] ath10k_pci 0000:01:00.0: [52]: 0x809BFC39 0x0040AEE4
> > 0x00424FE8 0x00000002
> > [ 131.748306] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
> > 0x00000004 0x004039D0
> > [ 131.857076] ieee80211 phy0: Hardware restart was requested
> > [ 131.862705] ath10k_pci 0000:01:00.0: failed to synchronize
> > monitor vdev 1 stop: -143
> > [ 131.870594] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
>
> [shafi] request to try with the latest backports and ath.git
> Also worth trying this fix in ath10k (once the next firmware version)
> is available. I will keep you posted. thank you !
>
> >
> > ~Benjamin
> >
> > On 11/30/2016 11:09 PM, Mohammed Shafi Shajakhan wrote:
> > >Hi,
> > >
> > >On Tue, Nov 29, 2016 at 11:22:12AM -0800, Benjamin Morgan wrote:
> > >>When we try to transmit traffic (ping) between two meshed ath10k
> > >>devices running latest lede we keep experiencing ath10k firmware
> > >>crashes. This seems to only happen when running in 802.11n/ac mode
> > >>but not in 802.11a/g mode. Also, from the station dumps it appears
> > >>that management traffic is flowing between the devices, however when
> > >>we try to send unicast data traffic the firmware crashes
> > >>immediately.
> > >[shafi] Did you get a chance to try with the below firmware as well
> > >https://github.com/kvalo/ath10k-firmware/blob/master/QCA988X/hw2.0/10.2.4.70/firmware-5.bin_10.2.4.70.58
> > >
> > >>Platform: Archer C7 AC1750 v2
> > >>Software Image: LEDE (HEAD, r2299) Commit: https://github.com/lede-project/source/commit/d596c21ebd5a3e6ce933eff3e51989031e4b1d58
> > >>
> > >>Crypto: wpa_supplicant
> > >>wpa_supplicant-wlan0.conf
> > >>network={
> > >>ssid="bmorgan_lede_mesh"
> > >>key_mgmt=SAE
> > >>mode=5
> > >>frequency=5180
> > >>psk="meshpassword"
> > >>}
> > >>
> > >>Backports Verstion:
> > >>[ 9.818007] Loading modules backported from Linux version
> > >>wt-2016-10-03-1-g6fcb1a6
> > >>[ 9.825736] Backport generated by backports.git
> > >>backports-20160324-9-g0e38f5c
> > >>
> > >>​​Ath10k Initialization on Station A (dmesg)
> > >>[ 9.896715] PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
> > >>[ 9.902622] ath10k_pci 0000:01:00.0: pci irq legacy oper_irq_mode
> > >>1 irq_mode 0 reset_mode 0
> > >>[ 10.123734] ath10k_pci 0000:01:00.0: Direct firmware load for
> > >>ath10k/pre-cal-pci-0000:01:00.0.bin failed with error -2
> > >>[ 10.134620] ath10k_pci 0000:01:00.0: Falling back to user helper
> > >>[ 10.287680] firmware ath10k!pre-cal-pci-0000:01:00.0.bin:
> > >>firmware_loading_store: map pages failed
> > >>[ 10.622789] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> > >>0x4100016c chip_id 0x043202ff sub 0000:0000
> > >>[ 10.632184] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
> > >>tracing 0 dfs 1 testmode 1
> > >>[ 10.645231] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
> > >>api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
> > >>[ 10.655660] ath10k_pci 0000:01:00.0: Direct firmware load for
> > >>ath10k/QCA988X/hw2.0/board-2.bin failed with error -2
> > >>[ 10.666264] ath10k_pci 0000:01:00.0: Falling back to user helper
> > >>[ 10.747925] firmware ath10k!QCA988X!hw2.0!board-2.bin:
> > >>firmware_loading_store: map pages failed
> > >>[ 11.011123] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> > >>crc32 bebc7c08
> > >>[ 12.155224] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
> > >>2 cal file max-sta 128 raw 0 hwcrypto 1
> > >>
> > >>Station A (wlan0):
> > >>18:A6:F7:23:6E:66
> > >>10.230.5.41
> > >>
> > >>Station B (wlan0):
> > >>18:a6:f7:26:0f:21
> > >>10.230.5.42
> > >>
> > >>Station Dump on Station A before ping:
> > >>Station 18:a6:f7:26:0f:21 (on wlan0)
> > >>inactive time:340 ms
> > >>rx bytes:2472
> > >>rx packets:28
> > >>tx bytes:1204
> > >>tx packets:9
> > >>tx retries:0
> > >>tx failed:0
> > >>rx drop misc:1
> > >>signal: -14 dBm
> > >>signal avg:-14 dBm
> > >>Toffset:18142530 us
> > >>tx bitrate:6.0 MBit/s
> > >>rx bitrate:6.0 MBit/s
> > >>rx duration:1524 us
> > >>mesh llid:0
> > >>mesh plid:0
> > >>mesh plink:ESTAB
> > >>mesh local PS mode:ACTIVE
> > >>mesh peer PS mode:UNKNOWN
> > >>mesh non-peer PS mode:ACTIVE
> > >>authorized:yes
> > >>authenticated:yes
> > >>associated:yes
> > >>preamble:long
> > >>WMM/WME:yes
> > >>MFP:yes
> > >>TDLS peer:no
> > >>DTIM period:2
> > >>beacon interval:1000
> > >>connected time:10 seconds
> > >>
> > >>​Crash Log on Station B (10.230.5.42)
> > >>[245.483888] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
> > >>2bab5ee9-08ff-4a17-95b1-636d212acebc)
> > >>[245.493020] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> > >>0x4100016c chip_id 0x043202ff sub 0000:0000
> > >>[245.502384] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
> > >>tracing 0 dfs 1 testmode 1
> > >>[245.515436] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api
> > >>5 features no-p2p,raw-mode,mfp crc32 9d340dd9
> > >>[245.525812] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> > >>crc32 bebc7c08
> > >>[245.533232] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
> > >>cal file max-sta 128 raw 0 hwcrypto 1
> > >>[245.544876] ath10k_pci 0000:01:00.0: firmware register dump:
> > >>[245.550633] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
> > >>0x009A4577 0x00955B31
> > >>[245.558676] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
> > >>0x00000002 0x00439E98
> > >>[245.566715] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
> > >>0x00407120 0x004436CC
> > >>[245.574749] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
> > >>0x009A3518 0x009A3526
> > >>[245.582793] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
> > >>0x00000000 0x00000000
> > >>[245.590836] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
> > >>0x0040AC60 0x0040AC09
> > >>[245.598882] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
> > >>0x00400000 0xC09A4577
> > >>[245.606923] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
> > >>0x0044110C 0x00442074
> > >>[245.614955] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
> > >>0x0044110C 0x00407120
> > >>[245.623000] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
> > >>0x0040AC14 0x00001580
> > >>[245.631043] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
> > >>0x009C643C 0x004436CC
> > >>[245.639086] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
> > >>0x004208FC 0x00439E4C
> > >>[245.647129] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
> > >>0x004208FC 0x00425404
> > >>[245.655170] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
> > >>0x00424FE8 0x00000002
> > >>[245.663198] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
> > >>0x00000004 0x004039D0
> > >>[245.767023] ieee80211 phy0: Hardware restart was requested
> > >>[245.772655] ath10k_pci 0000:01:00.0: failed to synchronize monitor
> > >>vdev 1 stop: -143
> > >>[245.780542] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
> > >>​
> > >>With wpa_supplicant turned off:
> > >>
> > >>Station Dump on Station A before ping:
> > >>Station 18:a6:f7:26:0f:21 (on wlan0)
> > >>inactive time:60 ms
> > >>rx bytes:3874
> > >>rx packets:67
> > >>tx bytes:707
> > >>tx packets:7
> > >>tx retries:0
> > >>tx failed:1
> > >>rx drop misc:3
> > >>signal: -14 dBm
> > >>signal avg:-14 dBm
> > >>Toffset:18446744073684250999 us
> > >>tx bitrate:6.0 MBit/s
> > >>rx bitrate:6.0 MBit/s
> > >>rx duration:1312 us
> > >>mesh llid:24953
> > >>mesh plid:59412
> > >>mesh plink:ESTAB
> > >>mesh local PS mode:ACTIVE
> > >>mesh peer PS mode:UNKNOWN
> > >>mesh non-peer PS mode:ACTIVE
> > >>authorized:yes
> > >>authenticated:yes
> > >>associated:yes
> > >>preamble:long
> > >>WMM/WME:yes
> > >>MFP:no
> > >>TDLS peer:no
> > >>DTIM period:2
> > >>beacon interval:1000
> > >>short slot time:yes
> > >>connected time:33 seconds
> > >>
> > >>Firmware crash on Station A (10.230.5.41)
> > >>[ 1040.984599] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
> > >>42d4f6dc-0e89-4505-aefb-58cbb70d0dce)
> > >>[ 1040.993732] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> > >>0x4100016c chip_id 0x043202ff sub 0000:0000
> > >>[ 1041.003100] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
> > >>tracing 0 dfs 1 testmode 1
> > >>[ 1041.016144] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
> > >>api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
> > >>[ 1041.026523] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> > >>crc32 bebc7c08
> > >>[ 1041.033940] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
> > >>2 cal file max-sta 128 raw 0 hwcrypto 1
> > >>[ 1041.045593] ath10k_pci 0000:01:00.0: firmware register dump:
> > >>[ 1041.051336] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
> > >>0x009A4577 0x00955B31
> > >>[ 1041.059383] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
> > >>0x00000001 0x00435354
> > >>[ 1041.067424] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
> > >>0x00407120 0x004436CC
> > >>[ 1041.075463] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
> > >>0x009A43A0 0x009A43DE
> > >>[ 1041.083490] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
> > >>0x00000000 0x00000000
> > >>[ 1041.091535] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
> > >>0x0040AC60 0x0040AC09
> > >>[ 1041.099579] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
> > >>0x00955A00 0xC09A4577
> > >>[ 1041.107622] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
> > >>0x0044110C 0x00442074
> > >>[ 1041.115661] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
> > >>0x0044110C 0x00407120
> > >>[ 1041.123689] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
> > >>0x0040AC10 0x00001580
> > >>[ 1041.131733] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
> > >>0x009C643C 0x004436CC
> > >>[ 1041.139777] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
> > >>0x004208FC 0x00439E4C
> > >>[ 1041.147820] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
> > >>0x004208FC 0x00425758
> > >>[ 1041.155860] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
> > >>0x00424FE8 0x00000002
> > >>[ 1041.163888] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
> > >>0x00000004 0x004039D0
> > >>[ 1041.267025] ieee80211 phy0: Hardware restart was requested
> > >>[ 1041.272656] ath10k_pci 0000:01:00.0: failed to synchronize
> > >>monitor vdev 1 stop: -143
> > >>[ 1041.280545] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
> > >>
> > >>​
> > >>Thanks in advance for your time and help,
> > >>
> > >>~Benjamin
> > >>
> >

2016-12-13 23:24:22

by Benjamin Morgan

[permalink] [raw]
Subject: Re: ath10k firmware crashes in mesh mode on QCA9880

Tested the 10.2.4.70.59-2 firmware and wpa_supplicant running WITHOUT
encryption and it still crashes. I suspect this means wpa_supplicant is
setting up the interface incorrectly and/or transmitting a malformed
packet that is causing the driver to crash.

[ 162.010206] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
d30144f6-a8fb-4c0d-bcdf-6ff3b2c37243)
[ 162.019322] ath10k_pci 0000:01:00.0: qca988x hw2.0 target 0x4100016c
chip_id 0x043202ff sub 0000:0000
[ 162.028687] ath10k_pci 0000:01:00.0: kconfig debug 1 debugfs 1
tracing 0 dfs 1 testmode 1
[ 162.041764] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.59-2 api
5 features no-p2p,raw-mode,mfp,allows-mesh-bcast crc32 4159f498
[ 162.053908] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
crc32 bebc7c08
[ 162.061332] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
cal file max-sta 128 raw 0 hwcrypto 1
[ 162.072975] ath10k_pci 0000:01:00.0: firmware register dump:
[ 162.078732] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
0x009A45AF 0x00955B31
[ 162.086771] ath10k_pci 0000:01:00.0: [04]: 0x009A45AF 0x00060130
0x00000001 0x40000000
[ 162.094804] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
0x00407120 0x004436CC
[ 162.102849] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
0x009A3550 0x009A355E
[ 162.110892] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
0x00000000 0x00000000
[ 162.118935] ath10k_pci 0000:01:00.0: [20]: 0x409A45AF 0x0040AAC4
0x0040AC60 0x0040AC09
[ 162.126978] ath10k_pci 0000:01:00.0: [24]: 0x809A44F2 0x0040AB24
0x00000000 0xC09A45AF
[ 162.135011] ath10k_pci 0000:01:00.0: [28]: 0x809A3A16 0x0040AB84
0x0044110C 0x00442074
[ 162.143056] ath10k_pci 0000:01:00.0: [32]: 0x809A601A 0x0040ABB4
0x0044110C 0x00407120
[ 162.151099] ath10k_pci 0000:01:00.0: [36]: 0x809A2EA4 0x0040ABF4
0x0040AC10 0x00001580
[ 162.159142] ath10k_pci 0000:01:00.0: [40]: 0x80990F63 0x0040AD04
0x009C6458 0x004436CC
[ 162.167185] ath10k_pci 0000:01:00.0: [44]: 0x80998520 0x0040AD64
0x004208FC 0x00439E4C
[ 162.175225] ath10k_pci 0000:01:00.0: [48]: 0x8099AEA5 0x0040AD84
0x004208FC 0x004265C4
[ 162.183253] ath10k_pci 0000:01:00.0: [52]: 0x809BFC39 0x0040AEE4
0x00424FE8 0x00000002
[ 162.191298] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
0x00000004 0x004039D0
[ 162.297229] ieee80211 phy0: Hardware restart was requested
[ 162.302880] ath10k_pci 0000:01:00.0: wmi disable pktlog

~Benjamin


On 12/13/2016 10:42 AM, Benjamin Morgan wrote:
> Just tested the latest 10.2.4.70.59-2 firmware and it still crashes
> with wpa_supplicant encrypted mesh =(
>
> [ 85.201440] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
> b7f44483-0488-46af-8dff-db88f4b56327)
> [ 85.210573] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> 0x4100016c chip_id 0x043202ff sub 0000:0000
> [ 85.219940] ath10k_pci 0000:01:00.0: kconfig debug 1 debugfs 1
> tracing 0 dfs 1 testmode 1
> [ 85.233034] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.59-2
> api 5 features no-p2p,raw-mode,mfp,allows-mesh-bcast crc32 4159f498
> [ 85.245177] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> crc32 bebc7c08
> [ 85.252592] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
> cal file max-sta 128 raw 0 hwcrypto 1
> [ 85.264235] ath10k_pci 0000:01:00.0: firmware register dump:
> [ 85.269992] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
> 0x009A45AF 0x00955B31
> [ 85.278031] ath10k_pci 0000:01:00.0: [04]: 0x009A45AF 0x00060130
> 0x00000002 0x00439E98
> [ 85.286078] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
> 0x00407120 0x004436CC
> [ 85.294107] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
> 0x009A3550 0x009A355E
> [ 85.302152] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
> 0x00000000 0x00000000
> [ 85.310195] ath10k_pci 0000:01:00.0: [20]: 0x409A45AF 0x0040AAC4
> 0x0040AC60 0x0040AC09
> [ 85.318239] ath10k_pci 0000:01:00.0: [24]: 0x809A44F2 0x0040AB24
> 0x00400000 0xC09A45AF
> [ 85.326282] ath10k_pci 0000:01:00.0: [28]: 0x809A3A16 0x0040AB84
> 0x0044110C 0x00442074
> [ 85.334314] ath10k_pci 0000:01:00.0: [32]: 0x809A601A 0x0040ABB4
> 0x0044110C 0x00407120
> [ 85.342350] ath10k_pci 0000:01:00.0: [36]: 0x809A2EA4 0x0040ABF4
> 0x0040AC14 0x00001580
> [ 85.350393] ath10k_pci 0000:01:00.0: [40]: 0x80990F63 0x0040AD04
> 0x009C6458 0x004436CC
> [ 85.358437] ath10k_pci 0000:01:00.0: [44]: 0x80998520 0x0040AD64
> 0x004208FC 0x00439E4C
> [ 85.366479] ath10k_pci 0000:01:00.0: [48]: 0x8099AEA5 0x0040AD84
> 0x004208FC 0x00425AAC
> [ 85.374512] ath10k_pci 0000:01:00.0: [52]: 0x809BFC39 0x0040AEE4
> 0x00424FE8 0x00000002
> [ 85.382548] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
> 0x00000004 0x004039D0
> [ 85.487067] ieee80211 phy0: Hardware restart was requested
> [ 85.492701] ath10k_pci 0000:01:00.0: wmi disable pktlog
>
> Any new leads on tracking down this issue?
>
> ~Benjamin
>
>
> On 12/06/2016 01:32 PM, Benjamin Morgan wrote:
>> 1. Yes, this appears to happens every time a unicast packet with
>> wpa_supplicant encryption in VHT80 mode is received. I haven't seen a
>> successful ping-pong pair.
>> 2. We tried with 10.2.4.70.42-2 firmware and still saw crashes.
>> 3. We ran our experiment again with extra debugging turned on.
>> Node A: 18:A6:F7:23:6E:66 | 10.230.5.41
>> Node B: 18:A6:F7:26:0F:21 | 10.230.5.42
>> The ping command we used was run on Node A was 'ping -s 1500 -i
>> 0.1 10.230.5.42'
>> Here is the dmesg log from Node B.
>>
>> [ 5413.478170] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5413.503954] ath10k_pci 0000:01:00.0: scan event bss channel type 4
>> reason 3 freq 5825 req_id 40961 scan_id 40960 vdev_id 0 state running
>> (2)
>> [ 5413.503985] ath10k_pci 0000:01:00.0: chan info err_code 0 freq
>> 5825 cmd_flags 1 noise_floor -105 rx_clear_count 7692807 cycle_count
>> 312271423
>> [ 5413.504029] ath10k_pci 0000:01:00.0: scan event completed type 2
>> reason 0 freq 5825 req_id 40961 scan_id 40960 vdev_id 0 state running
>> (2)
>> [ 5413.525868] ath10k_pci 0000:01:00.0: wmi vdev install key idx 1
>> cipher 4 len 16
>> [ 5413.526014] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 31
>> value 1
>> [ 5413.526193] ath10k_pci 0000:01:00.0: mac vdev 0 set keyidx 1
>> [ 5413.526216] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 31
>> value 1
>> [ 5413.526532] ath10k_pci 0000:01:00.0: mac chanctx add freq 5180
>> width 3 ptr 86db29b0
>> [ 5413.526556] ath10k_pci 0000:01:00.0: mac monitor recalc started? 0
>> needed? 0 allowed? 1
>> [ 5413.526574] ath10k_pci 0000:01:00.0: mac chanctx assign ptr
>> 86db29b0 vdev_id 0
>> [ 5413.526592] ath10k_pci 0000:01:00.0: mac vdev 0 start center_freq
>> 5180 phymode 11ac-vht80
>> [ 5413.526616] ath10k_pci 0000:01:00.0: wmi vdev start id 0x0 flags:
>> 0x0, freq 5180, mode 10, ch_flags: 0xA000000, max_power: 46
>> [ 5413.533099] ath10k_pci 0000:01:00.0: WMI_VDEV_START_RESP_EVENTID
>> [ 5413.533148] ath10k_pci 0000:01:00.0: mac vdev_id 0 txpower 23
>> [ 5413.533163] ath10k_pci 0000:01:00.0: mac txpower 23
>> [ 5413.533180] ath10k_pci 0000:01:00.0: wmi pdev set param 3 value 46
>> [ 5413.533247] ath10k_pci 0000:01:00.0: wmi pdev set param 4 value 46
>> [ 5413.533295] ath10k_pci 0000:01:00.0: mac chanctx change freq 5180
>> width 3 ptr 86db29b0 changed 10
>> [ 5413.533318] ath10k_pci 0000:01:00.0: mac chanctx change freq 5180
>> width 3 ptr 86db29b0 changed 2
>> [ 5413.533337] ath10k_pci 0000:01:00.0: mac monitor recalc started? 0
>> needed? 1 allowed? 1
>> [ 5413.533357] ath10k_pci 0000:01:00.0: WMI vdev create: id 1 type 4
>> subtype 0 macaddr 18:a6:f7:26:0f:21
>> [ 5413.533412] ath10k_pci 0000:01:00.0: mac monitor vdev 1 created
>> [ 5413.533463] ath10k_pci 0000:01:00.0: wmi vdev start id 0x1 flags:
>> 0x0, freq 5180, mode 10, ch_flags: 0xA000000, max_power: 46
>> [ 5413.937652] ath10k_pci 0000:01:00.0: wmi event debug mesg len 152
>> [ 5413.978273] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5414.478363] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5414.527015] ath10k_pci 0000:01:00.0: WMI_VDEV_START_RESP_EVENTID
>> [ 5414.527067] ath10k_pci 0000:01:00.0: wmi mgmt vdev up id 0x1 assoc
>> id 0 bssid 18:a6:f7:26:0f:21
>> [ 5414.527121] ath10k_pci 0000:01:00.0: mac monitor vdev 1 started
>> [ 5414.527165] ath10k_pci 0000:01:00.0: mac monitor started
>> [ 5414.527216] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 3
>> value 1000
>> [ 5414.527262] ath10k_pci 0000:01:00.0: mac vdev 0 beacon_interval 1000
>> [ 5414.527278] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
>> staggered
>> [ 5414.527294] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
>> [ 5414.527314] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
>> [ 5414.527330] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
>> value 2
>> [ 5414.527457] ath10k_pci 0000:01:00.0: wmi mgmt vdev up id 0x0 assoc
>> id 0 bssid 00:00:00:00:00:00
>> [ 5414.527501] ath10k_pci 0000:01:00.0: mac vdev 0 up
>> [ 5414.527564] ath10k_pci 0000:01:00.0: WMI_TBTTOFFSET_UPDATE_EVENTID
>> [ 5414.541090] ath10k_pci 0000:01:00.0: mac monitor recalc started? 1
>> needed? 1 allowed? 1
>> [ 5414.978454] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5415.478548] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5415.978649] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5416.445280] ath10k_pci 0000:01:00.0: mac monitor recalc started? 1
>> needed? 1 allowed? 1
>> [ 5416.478761] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5416.978879] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5417.478985] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5417.979081] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5418.479190] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5418.979301] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5419.479403] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5419.979551] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5420.479643] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5420.979746] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5421.479841] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5421.979940] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5422.480288] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5422.980386] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5423.480490] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5423.980600] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5424.480702] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5424.971969] ath10k_pci 0000:01:00.0: mac vdev 0 peer create
>> 18:a6:f7:23:6e:66 (new sta) sta 1 / 128 peer 2 / 144
>> [ 5424.972000] ath10k_pci 0000:01:00.0: wmi peer create vdev_id 0
>> peer_addr 18:a6:f7:23:6e:66
>> [ 5424.975107] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
>> staggered
>> [ 5424.975134] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
>> [ 5424.975219] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
>> [ 5424.975238] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
>> value 2
>> [ 5424.980787] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5425.204468] ath10k_pci 0000:01:00.0: mac sta 18:a6:f7:23:6e:66
>> associated
>> [ 5425.204531] ath10k_pci 0000:01:00.0: mac ht peer 18:a6:f7:23:6e:66
>> mcs cnt 24 nss 3
>> [ 5425.204548] ath10k_pci 0000:01:00.0: mac peer 18:a6:f7:23:6e:66 qos 1
>> [ 5425.204563] ath10k_pci 0000:01:00.0: mac peer 18:a6:f7:23:6e:66
>> phymode 11na-ht40
>> [ 5425.204585] ath10k_pci 0000:01:00.0: wmi peer assoc vdev 0 addr
>> 18:a6:f7:23:6e:66 (new)
>> [ 5425.204614] ath10k_pci 0000:01:00.0: wmi vdev 0 peer
>> 0x18:a6:f7:23:6e:66 set param 1 value 0
>> [ 5425.205376] ath10k_pci 0000:01:00.0: received event id 36891 not
>> implemented
>> [ 5425.209240] ath10k_pci 0000:01:00.0: wmi vdev install key idx 0
>> cipher 4 len 16
>> [ 5425.209655] ath10k_pci 0000:01:00.0: wmi vdev install key idx 1
>> cipher 4 len 16
>> [ 5425.209848] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 31
>> value 1
>> [ 5425.210196] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
>> staggered
>> [ 5425.210221] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
>> [ 5425.210296] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
>> [ 5425.210315] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
>> value 2
>> [ 5425.480863] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5425.938619] ath10k_pci 0000:01:00.0: wmi event debug mesg len 100
>> [ 5425.980946] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5425.995007] ath10k_pci 0000:01:00.0: mac sta rc update for
>> 18:a6:f7:23:6e:66 changed 00000001 bw 2 nss 3 smps 1
>> [ 5425.995060] ath10k_pci 0000:01:00.0: mac update sta
>> 18:a6:f7:23:6e:66 peer bw 2
>> [ 5425.995081] ath10k_pci 0000:01:00.0: wmi vdev 0 peer
>> 0x18:a6:f7:23:6e:66 set param 4 value 2
>> [ 5426.481030] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5426.981117] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5427.481206] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5427.981294] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5428.481628] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5428.981718] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5429.481812] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5429.981894] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5430.481985] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5430.982073] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5431.482174] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5431.982505] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5432.482597] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5432.982679] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5433.482765] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5433.982857] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5434.482946] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5434.983008] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5435.483100] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5435.983181] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5436.483276] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5436.983366] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5437.483445] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5437.983516] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5438.483607] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5438.983692] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 5439.439875] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
>> db76b67c-ca98-4519-a762-4ff4edb45526)
>> [ 5439.449007] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
>> 0x4100016c chip_id 0x043202ff sub 0000:0000
>> [ 5439.458378] ath10k_pci 0000:01:00.0: kconfig debug 1 debugfs 1
>> tracing 0 dfs 1 testmode 1
>> [ 5439.471460] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api
>> 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
>> [ 5439.481844] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
>> crc32 bebc7c08
>> [ 5439.489267] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
>> cal file max-sta 128 raw 0 hwcrypto 1
>> [ 5439.500918] ath10k_pci 0000:01:00.0: firmware register dump:
>> [ 5439.506678] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
>> 0x009A4577 0x00955B31
>> [ 5439.514706] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
>> 0x00000002 0x00439E98
>> [ 5439.522751] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
>> 0x00407120 0x004436CC
>> [ 5439.530794] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
>> 0x009A3518 0x009A3526
>> [ 5439.538834] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
>> 0x00000000 0x00000000
>> [ 5439.546871] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
>> 0x0040AC60 0x0040AC09
>> [ 5439.554915] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
>> 0x00400000 0xC09A4577
>> [ 5439.562948] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
>> 0x0044110C 0x00442074
>> [ 5439.570992] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
>> 0x0044110C 0x00407120
>> [ 5439.579032] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
>> 0x0040AC14 0x00001580
>> [ 5439.587070] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
>> 0x009C643C 0x004436CC
>> [ 5439.595113] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
>> 0x004208FC 0x00439E4C
>> [ 5439.603146] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
>> 0x004208FC 0x00425E00
>> [ 5439.611191] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
>> 0x00424FE8 0x00000002
>> [ 5439.619230] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
>> 0x00000004 0x004039D0
>> [ 5439.726818] ieee80211 phy0: Hardware restart was requested
>> [ 5439.732433] ath10k_pci 0000:01:00.0: wmi mgmt vdev down id 0x1
>> [ 5439.732461] ath10k_pci 0000:01:00.0: wmi vdev stop id 0x1
>> [ 5439.732482] ath10k_pci 0000:01:00.0: failed to synchronize monitor
>> vdev 1 stop: -143
>> [ 5439.740370] ath10k_pci 0000:01:00.0: mac monitor vdev 1 stopped
>> [ 5439.740386] ath10k_pci 0000:01:00.0: failed to stop monitor vdev:
>> -143
>> [ 5439.747042] ath10k_pci 0000:01:00.0: wmi disable pktlog
>>
>> We noticed in this log that when the radio starts up it says that it
>> is in VHT80 mode:
>> [ 5413.526592] ath10k_pci 0000:01:00.0: mac vdev 0 start center_freq
>> 5180 phymode 11ac-vht80
>>
>> But when a peer connects it seems to think the peer is in HT40 mode:
>> [ 5425.204563] ath10k_pci 0000:01:00.0: mac peer 18:a6:f7:23:6e:66
>> phymode 11na-ht40
>>
>> Compared to no encryption case - this log was taken from Node A:
>>
>> [ 24.874253] ath10k_pci 0000:01:00.0: mac chanctx add freq 5180
>> width 3 ptr 86d26db0
>> [ 24.874278] ath10k_pci 0000:01:00.0: mac monitor recalc started? 0
>> needed? 0 allowed? 1
>> [ 24.874296] ath10k_pci 0000:01:00.0: mac chanctx assign ptr
>> 86d26db0 vdev_id 0
>> [ 24.874312] ath10k_pci 0000:01:00.0: mac vdev 0 start center_freq
>> 5180 phymode 11ac-vht80
>> [ 24.874337] ath10k_pci 0000:01:00.0: wmi vdev start id 0x0 flags:
>> 0x0, freq 5180, mode 10, ch_flags: 0xA000000, max_power: 46
>> [ 24.881335] ath10k_pci 0000:01:00.0: WMI_VDEV_START_RESP_EVENTID
>> [ 24.881423] ath10k_pci 0000:01:00.0: mac vdev_id 0 txpower 23
>> [ 24.881438] ath10k_pci 0000:01:00.0: mac txpower 23
>> [ 24.881454] ath10k_pci 0000:01:00.0: wmi pdev set param 3 value 46
>> [ 24.881491] ath10k_pci 0000:01:00.0: wmi pdev set param 4 value 46
>> [ 24.881515] ath10k_pci 0000:01:00.0: mac chanctx change freq 5180
>> width 3 ptr 86d26db0 changed 10
>> [ 24.881535] ath10k_pci 0000:01:00.0: mac chanctx change freq 5180
>> width 3 ptr 86d26db0 changed 2
>> [ 24.881554] ath10k_pci 0000:01:00.0: mac monitor recalc started? 0
>> needed? 1 allowed? 1
>> [ 24.881574] ath10k_pci 0000:01:00.0: WMI vdev create: id 1 type 4
>> subtype 0 macaddr 18:a6:f7:23:6e:66
>> [ 24.881689] ath10k_pci 0000:01:00.0: mac monitor vdev 1 created
>> [ 24.881745] ath10k_pci 0000:01:00.0: wmi vdev start id 0x1 flags:
>> 0x0, freq 5180, mode 10, ch_flags: 0xA000000, max_power: 46
>> [ 25.273460] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 25.730570] ath10k_pci 0000:01:00.0: wmi event debug mesg len 300
>> [ 25.773566] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 25.874556] ath10k_pci 0000:01:00.0: WMI_VDEV_START_RESP_EVENTID
>> [ 25.879992] ath10k_pci 0000:01:00.0: wmi mgmt vdev up id 0x1 assoc
>> id 0 bssid 18:a6:f7:23:6e:66
>> [ 25.880077] ath10k_pci 0000:01:00.0: mac monitor vdev 1 started
>> [ 25.880093] ath10k_pci 0000:01:00.0: mac monitor started
>> [ 25.880139] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 3
>> value 1000
>> [ 25.880184] ath10k_pci 0000:01:00.0: mac vdev 0 beacon_interval 1000
>> [ 25.880199] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
>> staggered
>> [ 25.880215] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
>> [ 25.880235] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
>> [ 25.880250] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
>> value 2
>> [ 25.880988] ath10k_pci 0000:01:00.0: wmi mgmt vdev up id 0x0 assoc
>> id 0 bssid 00:00:00:00:00:00
>> [ 25.881035] ath10k_pci 0000:01:00.0: mac vdev 0 up
>> [ 25.881097] ath10k_pci 0000:01:00.0: WMI_TBTTOFFSET_UPDATE_EVENTID
>> [ 25.882968] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
>> [ 25.928796] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
>> staggered
>> [ 25.928821] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
>> [ 25.928866] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
>> [ 25.928883] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
>> value 2
>> [ 25.929020] ath10k_pci 0000:01:00.0: mac monitor recalc started? 1
>> needed? 1 allowed? 1
>> [ 25.941886] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
>> staggered
>> [ 25.941911] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
>> [ 25.941955] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
>> [ 25.941972] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
>> value 2
>> [ 25.953727] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
>> staggered
>> [ 25.953753] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
>> [ 25.953798] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
>> [ 25.953817] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
>> value 2
>> [ 25.970588] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
>> staggered
>> [ 25.970614] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
>> [ 25.970659] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
>> [ 25.970676] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
>> value 2
>> [ 25.989056] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
>> staggered
>> [ 25.989081] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
>> [ 25.989126] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
>> [ 25.989143] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
>> value 2
>> [ 26.071686] ath10k_pci 0000:01:00.0: mac vdev 0 peer create
>> 18:a6:f7:26:0f:21 (new sta) sta 1 / 128 peer 2 / 144
>> [ 26.071712] ath10k_pci 0000:01:00.0: wmi peer create vdev_id 0
>> peer_addr 18:a6:f7:26:0f:21
>> [ 26.071952] ath10k_pci 0000:01:00.0: mac sta 18:a6:f7:26:0f:21
>> associated
>> [ 26.071981] ath10k_pci 0000:01:00.0: mac ht peer 18:a6:f7:26:0f:21
>> mcs cnt 24 nss 3
>> [ 26.071999] ath10k_pci 0000:01:00.0: mac vht peer
>> 18:a6:f7:26:0f:21 max_mpdu 1048575 flags 0x601b001
>> [ 26.072013] ath10k_pci 0000:01:00.0: mac peer 18:a6:f7:26:0f:21 qos 1
>> [ 26.072028] ath10k_pci 0000:01:00.0: mac peer 18:a6:f7:26:0f:21
>> phymode 11ac-vht80
>> [ 26.072047] ath10k_pci 0000:01:00.0: wmi peer assoc vdev 0 addr
>> 18:a6:f7:26:0f:21 (new)
>> [ 26.072071] ath10k_pci 0000:01:00.0: wmi vdev 0 peer
>> 0x18:a6:f7:26:0f:21 set param 1 value 0
>> [ 26.072502] ath10k_pci 0000:01:00.0: received event id 36891 not
>> implemented
>> [ 26.074194] ath10k_pci 0000:01:00.0: mac sta rc update for
>> 18:a6:f7:26:0f:21 changed 00000000 bw 2 nss 3 smps 1
>> [ 26.074586] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
>> staggered
>> [ 26.074609] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
>> [ 26.074682] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
>> [ 26.074701] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13
>> value 2
>> [ 26.074760] ath10k_pci 0000:01:00.0: mac vdev 0 slot_time 2
>> [ 26.074779] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 7
>> value 2
>> [ 26.273652] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 26.730650] ath10k_pci 0000:01:00.0: wmi event debug mesg len 44
>> [ 26.773733] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>> [ 27.135445] ath10k_pci 0000:01:00.0: mac monitor recalc started? 1
>> needed? 1 allowed? 1
>> [ 27.273810] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
>>
>> It seems to start up in VHT80 mode and when it peers with Node B it
>> thinks Node B is also in VHT80 mode and ping works.
>>
>> 4. Beacons are sent at 6 Mb/s basic rate and unicast QoS Data is sent
>> with three spatial streams. Attached is the full pcap of the experiment.
>>
>> Thank you for looking into this!
>>
>> ~Benjamin
>>
>> On 12/05/2016 11:24 AM, Nagarajan, Ashok Raj wrote:
>>> 0x009A4577 0x00955B31
>>>
>>> Benjamin, Thanks for the logs.
>>> Quick questions to further debug the issue here,
>>>
>>> 1. Is this issue seen every time you start sending data traffic?
>>> 2. Issue seen with older firmwares? (FYR,
>>> http://linuxwireless.org/en/users/Drivers/ath10k/firmware/ )
>>> 3. Could you please share the dmesg from your device after enabling
>>> MAC and WMI logs in ath10k driver
>>> To enable debug logs please see
>>> http://linuxwireless.org/en/users/Drivers/ath10k/debug/
>>> 4. Do you know what is the Number of Spatial Streams seen in mesh
>>> beacons and in mesh data packet?
>>>
>>> Thanks,
>>> Ashok
>>
>

2016-12-13 23:54:50

by Rajkumar Manoharan

[permalink] [raw]
Subject: RE: ath10k firmware crashes in mesh mode on QCA9880

PiBUZXN0ZWQgdGhlIDEwLjIuNC43MC41OS0yIGZpcm13YXJlIGFuZCB3cGFfc3VwcGxpY2FudCBy
dW5uaW5nIFdJVEhPVVQNCj4gZW5jcnlwdGlvbiBhbmQgaXQgc3RpbGwgY3Jhc2hlcy4gSSBzdXNw
ZWN0IHRoaXMgbWVhbnMgd3BhX3N1cHBsaWNhbnQgaXMgc2V0dGluZyB1cA0KPiB0aGUgaW50ZXJm
YWNlIGluY29ycmVjdGx5IGFuZC9vciB0cmFuc21pdHRpbmcgYSBtYWxmb3JtZWQgcGFja2V0IHRo
YXQgaXMgY2F1c2luZw0KPiB0aGUgZHJpdmVyIHRvIGNyYXNoLg0KPiANCkJlbiwNCg0KSUlSQyBt
ZXNoIHN1cHBvcnQgd2FzIHZhbGlkYXRlZCBpbiBxY2E5ODh4IGluIFZIVCBtb2RlIHdoaWxlIGFn
by4gIEVpdGhlciBpdCBjb3VsZA0KYmUgcmVncmVzc2lvbiBpbiBkcml2ZXIvZncgb3IgbGVkZSBt
YWM4MDIxMSBwYWNrYWdlLg0KDQoxKSBDb3VsZCB5b3UgcGxlYXNlIHRyeSBwbGFpbiBiYWNrcG9y
dHMgaW4gbGVkZSB3L28gYXBwbHlpbmcgYXRoMTBrIHBhdGNoZXMuDQogICAgIEkgZG8gc2VlIDE2
ME1IeiBzdXBwb3J0IGluIExFREUuDQoyKSBUaGVyZSBhcmUgc29tZSBwZWVyIHN0YXRzIGR1bXAg
ZnJvbSB5b3VyIGVhcmxpZXIgbG9nLiBEaXNhYmxlIHBlZXIgc3RhdHMgDQogICAgIGJ5ICJwZWVy
X3N0YXRzIiBkZWJ1Z2ZzLg0KMykgUGxlYXNlIGNvbmZpcm0gdGhlIGJlaGF2aW9yIHdpdGggb2xk
ZXIgZmlybXdhcmUgcmV2aXNpb25zLg0KNCkgdXNlIGl3IHRvIGJyaW5nIHVwIG9wZW4gbWVzaCB0
byBydWxlIG91dCB3cGFfcyBjb25maWcNCg0KLVJhamt1bWFyDQoNCg==

2016-12-14 06:17:59

by Adrian Chadd

[permalink] [raw]
Subject: Re: ath10k firmware crashes in mesh mode on QCA9880

Hi!

ok, thanks! I've seen some .. annoying rate control related firmware
crashes if you aren't using 11ac / 11n rates (ie you're /really/
legacy, so I wondered if something similar is going on here.

Thanks!


-a


On 13 December 2016 at 22:06, Alexis Green <[email protected]> wrote:
> Hi Adrian,
>
> I have not done much testing of ath10k and ath9k devices in a single
> encrypted mesh recently, but I have a memory of only having this issue
> when communicating between ath10k devices.
>
> Alexis
>
> On Tue, Dec 13, 2016 at 9:53 PM, Adrian Chadd <[email protected]> wrote:
>> Hi!
>>
>> Hm! So is there a firmware bug if there are 11n only capable nodes in
>> an 11s mesh?
>>
>>
>>
>> -adrian

2016-12-05 19:24:16

by Ashok Raj Nagarajan

[permalink] [raw]
Subject: RE: ath10k firmware crashes in mesh mode on QCA9880

Pj4gQXBwbGllZCB0aGUgcGF0Y2ggYW5kIHRyaWVkIHdpdGggMTAuMi40LjcwLjU0IGZpcm13YXJl
IGFuZCBpdCBzdGlsbCBjcmFzaGVzOg0KDQo+PiBbICAxNDIuNDM4Mzc3XSBhdGgxMGtfcGNpIDAw
MDA6MDE6MDAuMDogZmlybXdhcmUgY3Jhc2hlZCEgKHV1aWQgDQo+PiBhNTQ5OTU4Mi1lMjIwLTQ2
ZDItOTM1OS0wYjQ0MjE5ZjY5ZWEpDQo+PiBbICAxNDIuNDQ3NTEyXSBhdGgxMGtfcGNpIDAwMDA6
MDE6MDAuMDogcWNhOTg4eCBodzIuMCB0YXJnZXQgMHg0MTAwMDE2YyANCj4+IGNoaXBfaWQgMHgw
NDMyMDJmZiBzdWIgMDAwMDowMDAwDQo+PiBbICAxNDIuNDU2ODc5XSBhdGgxMGtfcGNpIDAwMDA6
MDE6MDAuMDoga2NvbmZpZyBkZWJ1ZyAwIGRlYnVnZnMgMSANCj4+IHRyYWNpbmcgMCBkZnMgMSB0
ZXN0bW9kZSAxDQo+PiBbICAxNDIuNDY5OTE2XSBhdGgxMGtfcGNpIDAwMDA6MDE6MDAuMDogZmly
bXdhcmUgdmVyIDEwLjIuNC43MC41NCBhcGkgNSANCmZlYXR1cmVzIG5vLXAycCxyYXctbW9kZSxt
ZnAgY3JjMzIgOWQzNDBkZDkNCj4+IFsgIDE0Mi40ODAyOTVdIGF0aDEwa19wY2kgMDAwMDowMTow
MC4wOiBib2FyZF9maWxlIGFwaSAxIGJtaV9pZCBOL0EgDQo+PiBjcmMzMiBiZWJjN2MwOA0KPj4g
WyAgMTQyLjQ4NzcxN10gYXRoMTBrX3BjaSAwMDAwOjAxOjAwLjA6IGh0dC12ZXIgMi4xIHdtaS1v
cCA1IGh0dC1vcCAyIA0KPj4gY2FsIGZpbGUgbWF4LXN0YSAxMjggcmF3IDAgaHdjcnlwdG8gMQ0K
Pj4gWyAgMTQyLjQ5OTM2MV0gYXRoMTBrX3BjaSAwMDAwOjAxOjAwLjA6IGZpcm13YXJlIHJlZ2lz
dGVyIGR1bXA6DQo+PiBbICAxNDIuNTA1MTI0XSBhdGgxMGtfcGNpIDAwMDA6MDE6MDAuMDogWzAw
XTogMHg0MTAwMDE2QyAweDAwMDAxNUIzIA0KMHgwMDlBNDU3NyAweDAwOTU1QjMxDQoNCkJlbmph
bWluLCBUaGFua3MgZm9yIHRoZSBsb2dzLg0KUXVpY2sgcXVlc3Rpb25zIHRvIGZ1cnRoZXIgZGVi
dWcgdGhlIGlzc3VlIGhlcmUsDQoNCjEuIElzIHRoaXMgaXNzdWUgc2VlbiBldmVyeSB0aW1lIHlv
dSBzdGFydCBzZW5kaW5nIGRhdGEgdHJhZmZpYz8NCjIuIElzc3VlIHNlZW4gd2l0aCBvbGRlciBm
aXJtd2FyZXM/IChGWVIsIGh0dHA6Ly9saW51eHdpcmVsZXNzLm9yZy9lbi91c2Vycy9Ecml2ZXJz
L2F0aDEway9maXJtd2FyZS8gKQ0KMy4gQ291bGQgeW91IHBsZWFzZSBzaGFyZSB0aGUgZG1lc2cg
ZnJvbSB5b3VyIGRldmljZSBhZnRlciBlbmFibGluZyBNQUMgYW5kIFdNSSBsb2dzIGluIGF0aDEw
ayBkcml2ZXINCglUbyBlbmFibGUgZGVidWcgbG9ncyBwbGVhc2Ugc2VlIGh0dHA6Ly9saW51eHdp
cmVsZXNzLm9yZy9lbi91c2Vycy9Ecml2ZXJzL2F0aDEway9kZWJ1Zy8JDQo0LiBEbyB5b3Uga25v
dyB3aGF0IGlzIHRoZSBOdW1iZXIgb2YgU3BhdGlhbCBTdHJlYW1zIHNlZW4gaW4gbWVzaCBiZWFj
b25zIGFuZCBpbiBtZXNoIGRhdGEgcGFja2V0Pw0KDQpUaGFua3MsDQpBc2hvaw0K

2016-12-05 18:34:37

by Benjamin Morgan

[permalink] [raw]
Subject: Re: ath10k firmware crashes in mesh mode on QCA9880

Applied the patch and tried with 10.2.4.70.54 firmware and it still crashes:

[ 142.438377] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
a5499582-e220-46d2-9359-0b44219f69ea)
[ 142.447512] ath10k_pci 0000:01:00.0: qca988x hw2.0 target 0x4100016c
chip_id 0x043202ff sub 0000:0000
[ 142.456879] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
tracing 0 dfs 1 testmode 1
[ 142.469916] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api 5
features no-p2p,raw-mode,mfp crc32 9d340dd9
[ 142.480295] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
crc32 bebc7c08
[ 142.487717] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
cal file max-sta 128 raw 0 hwcrypto 1
[ 142.499361] ath10k_pci 0000:01:00.0: firmware register dump:
[ 142.505124] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
0x009A4577 0x00955B31
[ 142.513157] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
0x00000002 0x00439E98
[ 142.521203] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
0x00407120 0x004436CC
[ 142.529246] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
0x009A3518 0x009A3526
[ 142.537285] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x009A3EA6
0x00000000 0x00000000
[ 142.545324] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
0x0040AC60 0x0040AC09
[ 142.553356] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
0x00400000 0xC09A4577
[ 142.561400] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
0x0044110C 0x00442074
[ 142.569444] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
0x0044110C 0x00407120
[ 142.577483] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
0x0040AC14 0x00001580
[ 142.585522] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
0x009C643C 0x004436CC
[ 142.593554] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
0x004208FC 0x00439E4C
[ 142.601600] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
0x004208FC 0x0042638C
[ 142.609642] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
0x00424FE8 0x00000002
[ 142.617681] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
0x00000004 0x004039D0
[ 142.727220] ieee80211 phy0: Hardware restart was requested
[ 142.732850] ath10k_pci 0000:01:00.0: failed to synchronize monitor
vdev 1 stop: -143
[ 142.740739] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143


~Benjamin

On 12/03/2016 04:46 AM, Mohammed Shafi Shajakhan wrote:
> https://patchwork.kernel.org/patch/9437519/
> (sorry missed this in the previous thread)
>
> On Sat, Dec 03, 2016 at 06:13:58PM +0530, Mohammed Shafi Shajakhan wrote:
>> Hi Benjamin,
>>
>> On Fri, Dec 02, 2016 at 05:28:02PM -0800, Benjamin Morgan wrote:
>>> Just tried 10.2.4.70.58 firmware that you linked to and it still crashes:
>>>
>>> [ 131.568989] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
>>> 1838347e-9380-4a26-ac9d-2963ee95968b)
>>> [ 131.578124] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
>>> 0x4100016c chip_id 0x043202ff sub 0000:0000
>>> [ 131.587491] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
>>> tracing 0 dfs 1 testmode 1
>>> [ 131.600521] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.58
>>> api 5 features no-p2p,raw-mode,mfp crc32 e1af076f
>>> [ 131.610899] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
>>> crc32 bebc7c08
>>> [ 131.618325] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
>>> 2 cal file max-sta 128 raw 0 hwcrypto 1
>>> [ 131.629965] ath10k_pci 0000:01:00.0: firmware register dump:
>>> [ 131.635728] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
>>> 0x009A45AF 0x00955B31
>>> [ 131.643761] ath10k_pci 0000:01:00.0: [04]: 0x009A45AF 0x00060130
>>> 0x00000002 0x00439E98
>>> [ 131.651806] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
>>> 0x00407120 0x004436CC
>>> [ 131.659852] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
>>> 0x009A3550 0x009A355E
>>> [ 131.667892] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x009A31D6
>>> 0x00000000 0x00000000
>>> [ 131.675936] ath10k_pci 0000:01:00.0: [20]: 0x409A45AF 0x0040AAC4
>>> 0x0040AC60 0x0040AC09
>>> [ 131.683968] ath10k_pci 0000:01:00.0: [24]: 0x809A44F2 0x0040AB24
>>> 0x00400000 0xC09A45AF
>>> [ 131.692013] ath10k_pci 0000:01:00.0: [28]: 0x809A3A16 0x0040AB84
>>> 0x0044110C 0x00442074
>>> [ 131.700056] ath10k_pci 0000:01:00.0: [32]: 0x809A601A 0x0040ABB4
>>> 0x0044110C 0x00407120
>>> [ 131.708100] ath10k_pci 0000:01:00.0: [36]: 0x809A2EA4 0x0040ABF4
>>> 0x0040AC14 0x00001580
>>> [ 131.716143] ath10k_pci 0000:01:00.0: [40]: 0x80990F63 0x0040AD04
>>> 0x009C6458 0x004436CC
>>> [ 131.724175] ath10k_pci 0000:01:00.0: [44]: 0x80998520 0x0040AD64
>>> 0x004208FC 0x00439E4C
>>> [ 131.732220] ath10k_pci 0000:01:00.0: [48]: 0x8099AEA5 0x0040AD84
>>> 0x004208FC 0x00425874
>>> [ 131.740263] ath10k_pci 0000:01:00.0: [52]: 0x809BFC39 0x0040AEE4
>>> 0x00424FE8 0x00000002
>>> [ 131.748306] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
>>> 0x00000004 0x004039D0
>>> [ 131.857076] ieee80211 phy0: Hardware restart was requested
>>> [ 131.862705] ath10k_pci 0000:01:00.0: failed to synchronize
>>> monitor vdev 1 stop: -143
>>> [ 131.870594] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
>> [shafi] request to try with the latest backports and ath.git
>> Also worth trying this fix in ath10k (once the next firmware version)
>> is available. I will keep you posted. thank you !
>>
>>> ~Benjamin
>>>
>>> On 11/30/2016 11:09 PM, Mohammed Shafi Shajakhan wrote:
>>>> Hi,
>>>>
>>>> On Tue, Nov 29, 2016 at 11:22:12AM -0800, Benjamin Morgan wrote:
>>>>> When we try to transmit traffic (ping) between two meshed ath10k
>>>>> devices running latest lede we keep experiencing ath10k firmware
>>>>> crashes. This seems to only happen when running in 802.11n/ac mode
>>>>> but not in 802.11a/g mode. Also, from the station dumps it appears
>>>>> that management traffic is flowing between the devices, however when
>>>>> we try to send unicast data traffic the firmware crashes
>>>>> immediately.
>>>> [shafi] Did you get a chance to try with the below firmware as well
>>>> https://github.com/kvalo/ath10k-firmware/blob/master/QCA988X/hw2.0/10.2.4.70/firmware-5.bin_10.2.4.70.58
>>>>
>>>>> Platform: Archer C7 AC1750 v2
>>>>> Software Image: LEDE (HEAD, r2299) Commit: https://github.com/lede-project/source/commit/d596c21ebd5a3e6ce933eff3e51989031e4b1d58
>>>>>
>>>>> Crypto: wpa_supplicant
>>>>> wpa_supplicant-wlan0.conf
>>>>> network={
>>>>> ssid="bmorgan_lede_mesh"
>>>>> key_mgmt=SAE
>>>>> mode=5
>>>>> frequency=5180
>>>>> psk="meshpassword"
>>>>> }
>>>>>
>>>>> Backports Verstion:
>>>>> [ 9.818007] Loading modules backported from Linux version
>>>>> wt-2016-10-03-1-g6fcb1a6
>>>>> [ 9.825736] Backport generated by backports.git
>>>>> backports-20160324-9-g0e38f5c
>>>>>
>>>>> ​​Ath10k Initialization on Station A (dmesg)
>>>>> [ 9.896715] PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
>>>>> [ 9.902622] ath10k_pci 0000:01:00.0: pci irq legacy oper_irq_mode
>>>>> 1 irq_mode 0 reset_mode 0
>>>>> [ 10.123734] ath10k_pci 0000:01:00.0: Direct firmware load for
>>>>> ath10k/pre-cal-pci-0000:01:00.0.bin failed with error -2
>>>>> [ 10.134620] ath10k_pci 0000:01:00.0: Falling back to user helper
>>>>> [ 10.287680] firmware ath10k!pre-cal-pci-0000:01:00.0.bin:
>>>>> firmware_loading_store: map pages failed
>>>>> [ 10.622789] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
>>>>> 0x4100016c chip_id 0x043202ff sub 0000:0000
>>>>> [ 10.632184] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
>>>>> tracing 0 dfs 1 testmode 1
>>>>> [ 10.645231] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
>>>>> api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
>>>>> [ 10.655660] ath10k_pci 0000:01:00.0: Direct firmware load for
>>>>> ath10k/QCA988X/hw2.0/board-2.bin failed with error -2
>>>>> [ 10.666264] ath10k_pci 0000:01:00.0: Falling back to user helper
>>>>> [ 10.747925] firmware ath10k!QCA988X!hw2.0!board-2.bin:
>>>>> firmware_loading_store: map pages failed
>>>>> [ 11.011123] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
>>>>> crc32 bebc7c08
>>>>> [ 12.155224] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
>>>>> 2 cal file max-sta 128 raw 0 hwcrypto 1
>>>>>
>>>>> Station A (wlan0):
>>>>> 18:A6:F7:23:6E:66
>>>>> 10.230.5.41
>>>>>
>>>>> Station B (wlan0):
>>>>> 18:a6:f7:26:0f:21
>>>>> 10.230.5.42
>>>>>
>>>>> Station Dump on Station A before ping:
>>>>> Station 18:a6:f7:26:0f:21 (on wlan0)
>>>>> inactive time:340 ms
>>>>> rx bytes:2472
>>>>> rx packets:28
>>>>> tx bytes:1204
>>>>> tx packets:9
>>>>> tx retries:0
>>>>> tx failed:0
>>>>> rx drop misc:1
>>>>> signal: -14 dBm
>>>>> signal avg:-14 dBm
>>>>> Toffset:18142530 us
>>>>> tx bitrate:6.0 MBit/s
>>>>> rx bitrate:6.0 MBit/s
>>>>> rx duration:1524 us
>>>>> mesh llid:0
>>>>> mesh plid:0
>>>>> mesh plink:ESTAB
>>>>> mesh local PS mode:ACTIVE
>>>>> mesh peer PS mode:UNKNOWN
>>>>> mesh non-peer PS mode:ACTIVE
>>>>> authorized:yes
>>>>> authenticated:yes
>>>>> associated:yes
>>>>> preamble:long
>>>>> WMM/WME:yes
>>>>> MFP:yes
>>>>> TDLS peer:no
>>>>> DTIM period:2
>>>>> beacon interval:1000
>>>>> connected time:10 seconds
>>>>>
>>>>> ​Crash Log on Station B (10.230.5.42)
>>>>> [245.483888] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
>>>>> 2bab5ee9-08ff-4a17-95b1-636d212acebc)
>>>>> [245.493020] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
>>>>> 0x4100016c chip_id 0x043202ff sub 0000:0000
>>>>> [245.502384] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
>>>>> tracing 0 dfs 1 testmode 1
>>>>> [245.515436] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api
>>>>> 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
>>>>> [245.525812] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
>>>>> crc32 bebc7c08
>>>>> [245.533232] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
>>>>> cal file max-sta 128 raw 0 hwcrypto 1
>>>>> [245.544876] ath10k_pci 0000:01:00.0: firmware register dump:
>>>>> [245.550633] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
>>>>> 0x009A4577 0x00955B31
>>>>> [245.558676] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
>>>>> 0x00000002 0x00439E98
>>>>> [245.566715] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
>>>>> 0x00407120 0x004436CC
>>>>> [245.574749] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
>>>>> 0x009A3518 0x009A3526
>>>>> [245.582793] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
>>>>> 0x00000000 0x00000000
>>>>> [245.590836] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
>>>>> 0x0040AC60 0x0040AC09
>>>>> [245.598882] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
>>>>> 0x00400000 0xC09A4577
>>>>> [245.606923] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
>>>>> 0x0044110C 0x00442074
>>>>> [245.614955] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
>>>>> 0x0044110C 0x00407120
>>>>> [245.623000] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
>>>>> 0x0040AC14 0x00001580
>>>>> [245.631043] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
>>>>> 0x009C643C 0x004436CC
>>>>> [245.639086] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
>>>>> 0x004208FC 0x00439E4C
>>>>> [245.647129] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
>>>>> 0x004208FC 0x00425404
>>>>> [245.655170] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
>>>>> 0x00424FE8 0x00000002
>>>>> [245.663198] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
>>>>> 0x00000004 0x004039D0
>>>>> [245.767023] ieee80211 phy0: Hardware restart was requested
>>>>> [245.772655] ath10k_pci 0000:01:00.0: failed to synchronize monitor
>>>>> vdev 1 stop: -143
>>>>> [245.780542] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
>>>>> ​
>>>>> With wpa_supplicant turned off:
>>>>>
>>>>> Station Dump on Station A before ping:
>>>>> Station 18:a6:f7:26:0f:21 (on wlan0)
>>>>> inactive time:60 ms
>>>>> rx bytes:3874
>>>>> rx packets:67
>>>>> tx bytes:707
>>>>> tx packets:7
>>>>> tx retries:0
>>>>> tx failed:1
>>>>> rx drop misc:3
>>>>> signal: -14 dBm
>>>>> signal avg:-14 dBm
>>>>> Toffset:18446744073684250999 us
>>>>> tx bitrate:6.0 MBit/s
>>>>> rx bitrate:6.0 MBit/s
>>>>> rx duration:1312 us
>>>>> mesh llid:24953
>>>>> mesh plid:59412
>>>>> mesh plink:ESTAB
>>>>> mesh local PS mode:ACTIVE
>>>>> mesh peer PS mode:UNKNOWN
>>>>> mesh non-peer PS mode:ACTIVE
>>>>> authorized:yes
>>>>> authenticated:yes
>>>>> associated:yes
>>>>> preamble:long
>>>>> WMM/WME:yes
>>>>> MFP:no
>>>>> TDLS peer:no
>>>>> DTIM period:2
>>>>> beacon interval:1000
>>>>> short slot time:yes
>>>>> connected time:33 seconds
>>>>>
>>>>> Firmware crash on Station A (10.230.5.41)
>>>>> [ 1040.984599] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
>>>>> 42d4f6dc-0e89-4505-aefb-58cbb70d0dce)
>>>>> [ 1040.993732] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
>>>>> 0x4100016c chip_id 0x043202ff sub 0000:0000
>>>>> [ 1041.003100] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
>>>>> tracing 0 dfs 1 testmode 1
>>>>> [ 1041.016144] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
>>>>> api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
>>>>> [ 1041.026523] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
>>>>> crc32 bebc7c08
>>>>> [ 1041.033940] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
>>>>> 2 cal file max-sta 128 raw 0 hwcrypto 1
>>>>> [ 1041.045593] ath10k_pci 0000:01:00.0: firmware register dump:
>>>>> [ 1041.051336] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
>>>>> 0x009A4577 0x00955B31
>>>>> [ 1041.059383] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
>>>>> 0x00000001 0x00435354
>>>>> [ 1041.067424] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
>>>>> 0x00407120 0x004436CC
>>>>> [ 1041.075463] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
>>>>> 0x009A43A0 0x009A43DE
>>>>> [ 1041.083490] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
>>>>> 0x00000000 0x00000000
>>>>> [ 1041.091535] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
>>>>> 0x0040AC60 0x0040AC09
>>>>> [ 1041.099579] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
>>>>> 0x00955A00 0xC09A4577
>>>>> [ 1041.107622] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
>>>>> 0x0044110C 0x00442074
>>>>> [ 1041.115661] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
>>>>> 0x0044110C 0x00407120
>>>>> [ 1041.123689] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
>>>>> 0x0040AC10 0x00001580
>>>>> [ 1041.131733] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
>>>>> 0x009C643C 0x004436CC
>>>>> [ 1041.139777] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
>>>>> 0x004208FC 0x00439E4C
>>>>> [ 1041.147820] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
>>>>> 0x004208FC 0x00425758
>>>>> [ 1041.155860] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
>>>>> 0x00424FE8 0x00000002
>>>>> [ 1041.163888] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
>>>>> 0x00000004 0x004039D0
>>>>> [ 1041.267025] ieee80211 phy0: Hardware restart was requested
>>>>> [ 1041.272656] ath10k_pci 0000:01:00.0: failed to synchronize
>>>>> monitor vdev 1 stop: -143
>>>>> [ 1041.280545] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
>>>>>
>>>>> ​
>>>>> Thanks in advance for your time and help,
>>>>>
>>>>> ~Benjamin
>>>>>

2016-12-06 21:32:47

by Benjamin Morgan

[permalink] [raw]
Subject: Re: ath10k firmware crashes in mesh mode on QCA9880

1. Yes, this appears to happens every time a unicast packet with
wpa_supplicant encryption in VHT80 mode is received. I haven't seen a
successful ping-pong pair.
2. We tried with 10.2.4.70.42-2 firmware and still saw crashes.
3. We ran our experiment again with extra debugging turned on.
Node A: 18:A6:F7:23:6E:66 | 10.230.5.41
Node B: 18:A6:F7:26:0F:21 | 10.230.5.42
The ping command we used was run on Node A was 'ping -s 1500 -i 0.1
10.230.5.42'
Here is the dmesg log from Node B.

[ 5413.478170] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5413.503954] ath10k_pci 0000:01:00.0: scan event bss channel type 4
reason 3 freq 5825 req_id 40961 scan_id 40960 vdev_id 0 state running (2)
[ 5413.503985] ath10k_pci 0000:01:00.0: chan info err_code 0 freq 5825
cmd_flags 1 noise_floor -105 rx_clear_count 7692807 cycle_count 312271423
[ 5413.504029] ath10k_pci 0000:01:00.0: scan event completed type 2
reason 0 freq 5825 req_id 40961 scan_id 40960 vdev_id 0 state running (2)
[ 5413.525868] ath10k_pci 0000:01:00.0: wmi vdev install key idx 1
cipher 4 len 16
[ 5413.526014] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 31 value 1
[ 5413.526193] ath10k_pci 0000:01:00.0: mac vdev 0 set keyidx 1
[ 5413.526216] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 31 value 1
[ 5413.526532] ath10k_pci 0000:01:00.0: mac chanctx add freq 5180 width
3 ptr 86db29b0
[ 5413.526556] ath10k_pci 0000:01:00.0: mac monitor recalc started? 0
needed? 0 allowed? 1
[ 5413.526574] ath10k_pci 0000:01:00.0: mac chanctx assign ptr 86db29b0
vdev_id 0
[ 5413.526592] ath10k_pci 0000:01:00.0: mac vdev 0 start center_freq
5180 phymode 11ac-vht80
[ 5413.526616] ath10k_pci 0000:01:00.0: wmi vdev start id 0x0 flags:
0x0, freq 5180, mode 10, ch_flags: 0xA000000, max_power: 46
[ 5413.533099] ath10k_pci 0000:01:00.0: WMI_VDEV_START_RESP_EVENTID
[ 5413.533148] ath10k_pci 0000:01:00.0: mac vdev_id 0 txpower 23
[ 5413.533163] ath10k_pci 0000:01:00.0: mac txpower 23
[ 5413.533180] ath10k_pci 0000:01:00.0: wmi pdev set param 3 value 46
[ 5413.533247] ath10k_pci 0000:01:00.0: wmi pdev set param 4 value 46
[ 5413.533295] ath10k_pci 0000:01:00.0: mac chanctx change freq 5180
width 3 ptr 86db29b0 changed 10
[ 5413.533318] ath10k_pci 0000:01:00.0: mac chanctx change freq 5180
width 3 ptr 86db29b0 changed 2
[ 5413.533337] ath10k_pci 0000:01:00.0: mac monitor recalc started? 0
needed? 1 allowed? 1
[ 5413.533357] ath10k_pci 0000:01:00.0: WMI vdev create: id 1 type 4
subtype 0 macaddr 18:a6:f7:26:0f:21
[ 5413.533412] ath10k_pci 0000:01:00.0: mac monitor vdev 1 created
[ 5413.533463] ath10k_pci 0000:01:00.0: wmi vdev start id 0x1 flags:
0x0, freq 5180, mode 10, ch_flags: 0xA000000, max_power: 46
[ 5413.937652] ath10k_pci 0000:01:00.0: wmi event debug mesg len 152
[ 5413.978273] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5414.478363] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5414.527015] ath10k_pci 0000:01:00.0: WMI_VDEV_START_RESP_EVENTID
[ 5414.527067] ath10k_pci 0000:01:00.0: wmi mgmt vdev up id 0x1 assoc id
0 bssid 18:a6:f7:26:0f:21
[ 5414.527121] ath10k_pci 0000:01:00.0: mac monitor vdev 1 started
[ 5414.527165] ath10k_pci 0000:01:00.0: mac monitor started
[ 5414.527216] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 3
value 1000
[ 5414.527262] ath10k_pci 0000:01:00.0: mac vdev 0 beacon_interval 1000
[ 5414.527278] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
staggered
[ 5414.527294] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
[ 5414.527314] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
[ 5414.527330] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13 value 2
[ 5414.527457] ath10k_pci 0000:01:00.0: wmi mgmt vdev up id 0x0 assoc id
0 bssid 00:00:00:00:00:00
[ 5414.527501] ath10k_pci 0000:01:00.0: mac vdev 0 up
[ 5414.527564] ath10k_pci 0000:01:00.0: WMI_TBTTOFFSET_UPDATE_EVENTID
[ 5414.541090] ath10k_pci 0000:01:00.0: mac monitor recalc started? 1
needed? 1 allowed? 1
[ 5414.978454] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5415.478548] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5415.978649] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5416.445280] ath10k_pci 0000:01:00.0: mac monitor recalc started? 1
needed? 1 allowed? 1
[ 5416.478761] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5416.978879] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5417.478985] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5417.979081] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5418.479190] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5418.979301] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5419.479403] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5419.979551] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5420.479643] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5420.979746] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5421.479841] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5421.979940] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5422.480288] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5422.980386] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5423.480490] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5423.980600] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5424.480702] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5424.971969] ath10k_pci 0000:01:00.0: mac vdev 0 peer create
18:a6:f7:23:6e:66 (new sta) sta 1 / 128 peer 2 / 144
[ 5424.972000] ath10k_pci 0000:01:00.0: wmi peer create vdev_id 0
peer_addr 18:a6:f7:23:6e:66
[ 5424.975107] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
staggered
[ 5424.975134] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
[ 5424.975219] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
[ 5424.975238] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13 value 2
[ 5424.980787] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5425.204468] ath10k_pci 0000:01:00.0: mac sta 18:a6:f7:23:6e:66 associated
[ 5425.204531] ath10k_pci 0000:01:00.0: mac ht peer 18:a6:f7:23:6e:66
mcs cnt 24 nss 3
[ 5425.204548] ath10k_pci 0000:01:00.0: mac peer 18:a6:f7:23:6e:66 qos 1
[ 5425.204563] ath10k_pci 0000:01:00.0: mac peer 18:a6:f7:23:6e:66
phymode 11na-ht40
[ 5425.204585] ath10k_pci 0000:01:00.0: wmi peer assoc vdev 0 addr
18:a6:f7:23:6e:66 (new)
[ 5425.204614] ath10k_pci 0000:01:00.0: wmi vdev 0 peer
0x18:a6:f7:23:6e:66 set param 1 value 0
[ 5425.205376] ath10k_pci 0000:01:00.0: received event id 36891 not
implemented
[ 5425.209240] ath10k_pci 0000:01:00.0: wmi vdev install key idx 0
cipher 4 len 16
[ 5425.209655] ath10k_pci 0000:01:00.0: wmi vdev install key idx 1
cipher 4 len 16
[ 5425.209848] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 31 value 1
[ 5425.210196] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
staggered
[ 5425.210221] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
[ 5425.210296] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
[ 5425.210315] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13 value 2
[ 5425.480863] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5425.938619] ath10k_pci 0000:01:00.0: wmi event debug mesg len 100
[ 5425.980946] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5425.995007] ath10k_pci 0000:01:00.0: mac sta rc update for
18:a6:f7:23:6e:66 changed 00000001 bw 2 nss 3 smps 1
[ 5425.995060] ath10k_pci 0000:01:00.0: mac update sta 18:a6:f7:23:6e:66
peer bw 2
[ 5425.995081] ath10k_pci 0000:01:00.0: wmi vdev 0 peer
0x18:a6:f7:23:6e:66 set param 4 value 2
[ 5426.481030] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5426.981117] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5427.481206] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5427.981294] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5428.481628] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5428.981718] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5429.481812] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5429.981894] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5430.481985] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5430.982073] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5431.482174] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5431.982505] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5432.482597] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5432.982679] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5433.482765] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5433.982857] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5434.482946] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5434.983008] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5435.483100] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5435.983181] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5436.483276] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5436.983366] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5437.483445] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5437.983516] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5438.483607] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5438.983692] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 5439.439875] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
db76b67c-ca98-4519-a762-4ff4edb45526)
[ 5439.449007] ath10k_pci 0000:01:00.0: qca988x hw2.0 target 0x4100016c
chip_id 0x043202ff sub 0000:0000
[ 5439.458378] ath10k_pci 0000:01:00.0: kconfig debug 1 debugfs 1
tracing 0 dfs 1 testmode 1
[ 5439.471460] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api 5
features no-p2p,raw-mode,mfp crc32 9d340dd9
[ 5439.481844] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
crc32 bebc7c08
[ 5439.489267] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
cal file max-sta 128 raw 0 hwcrypto 1
[ 5439.500918] ath10k_pci 0000:01:00.0: firmware register dump:
[ 5439.506678] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
0x009A4577 0x00955B31
[ 5439.514706] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
0x00000002 0x00439E98
[ 5439.522751] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
0x00407120 0x004436CC
[ 5439.530794] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
0x009A3518 0x009A3526
[ 5439.538834] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
0x00000000 0x00000000
[ 5439.546871] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
0x0040AC60 0x0040AC09
[ 5439.554915] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
0x00400000 0xC09A4577
[ 5439.562948] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
0x0044110C 0x00442074
[ 5439.570992] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
0x0044110C 0x00407120
[ 5439.579032] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
0x0040AC14 0x00001580
[ 5439.587070] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
0x009C643C 0x004436CC
[ 5439.595113] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
0x004208FC 0x00439E4C
[ 5439.603146] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
0x004208FC 0x00425E00
[ 5439.611191] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
0x00424FE8 0x00000002
[ 5439.619230] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
0x00000004 0x004039D0
[ 5439.726818] ieee80211 phy0: Hardware restart was requested
[ 5439.732433] ath10k_pci 0000:01:00.0: wmi mgmt vdev down id 0x1
[ 5439.732461] ath10k_pci 0000:01:00.0: wmi vdev stop id 0x1
[ 5439.732482] ath10k_pci 0000:01:00.0: failed to synchronize monitor
vdev 1 stop: -143
[ 5439.740370] ath10k_pci 0000:01:00.0: mac monitor vdev 1 stopped
[ 5439.740386] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
[ 5439.747042] ath10k_pci 0000:01:00.0: wmi disable pktlog

We noticed in this log that when the radio starts up it says that it is
in VHT80 mode:
[ 5413.526592] ath10k_pci 0000:01:00.0: mac vdev 0 start center_freq
5180 phymode 11ac-vht80

But when a peer connects it seems to think the peer is in HT40 mode:
[ 5425.204563] ath10k_pci 0000:01:00.0: mac peer 18:a6:f7:23:6e:66
phymode 11na-ht40

Compared to no encryption case - this log was taken from Node A:

[ 24.874253] ath10k_pci 0000:01:00.0: mac chanctx add freq 5180 width
3 ptr 86d26db0
[ 24.874278] ath10k_pci 0000:01:00.0: mac monitor recalc started? 0
needed? 0 allowed? 1
[ 24.874296] ath10k_pci 0000:01:00.0: mac chanctx assign ptr 86d26db0
vdev_id 0
[ 24.874312] ath10k_pci 0000:01:00.0: mac vdev 0 start center_freq
5180 phymode 11ac-vht80
[ 24.874337] ath10k_pci 0000:01:00.0: wmi vdev start id 0x0 flags:
0x0, freq 5180, mode 10, ch_flags: 0xA000000, max_power: 46
[ 24.881335] ath10k_pci 0000:01:00.0: WMI_VDEV_START_RESP_EVENTID
[ 24.881423] ath10k_pci 0000:01:00.0: mac vdev_id 0 txpower 23
[ 24.881438] ath10k_pci 0000:01:00.0: mac txpower 23
[ 24.881454] ath10k_pci 0000:01:00.0: wmi pdev set param 3 value 46
[ 24.881491] ath10k_pci 0000:01:00.0: wmi pdev set param 4 value 46
[ 24.881515] ath10k_pci 0000:01:00.0: mac chanctx change freq 5180
width 3 ptr 86d26db0 changed 10
[ 24.881535] ath10k_pci 0000:01:00.0: mac chanctx change freq 5180
width 3 ptr 86d26db0 changed 2
[ 24.881554] ath10k_pci 0000:01:00.0: mac monitor recalc started? 0
needed? 1 allowed? 1
[ 24.881574] ath10k_pci 0000:01:00.0: WMI vdev create: id 1 type 4
subtype 0 macaddr 18:a6:f7:23:6e:66
[ 24.881689] ath10k_pci 0000:01:00.0: mac monitor vdev 1 created
[ 24.881745] ath10k_pci 0000:01:00.0: wmi vdev start id 0x1 flags:
0x0, freq 5180, mode 10, ch_flags: 0xA000000, max_power: 46
[ 25.273460] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 25.730570] ath10k_pci 0000:01:00.0: wmi event debug mesg len 300
[ 25.773566] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 25.874556] ath10k_pci 0000:01:00.0: WMI_VDEV_START_RESP_EVENTID
[ 25.879992] ath10k_pci 0000:01:00.0: wmi mgmt vdev up id 0x1 assoc id
0 bssid 18:a6:f7:23:6e:66
[ 25.880077] ath10k_pci 0000:01:00.0: mac monitor vdev 1 started
[ 25.880093] ath10k_pci 0000:01:00.0: mac monitor started
[ 25.880139] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 3
value 1000
[ 25.880184] ath10k_pci 0000:01:00.0: mac vdev 0 beacon_interval 1000
[ 25.880199] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
staggered
[ 25.880215] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
[ 25.880235] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
[ 25.880250] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13 value 2
[ 25.880988] ath10k_pci 0000:01:00.0: wmi mgmt vdev up id 0x0 assoc id
0 bssid 00:00:00:00:00:00
[ 25.881035] ath10k_pci 0000:01:00.0: mac vdev 0 up
[ 25.881097] ath10k_pci 0000:01:00.0: WMI_TBTTOFFSET_UPDATE_EVENTID
[ 25.882968] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[ 25.928796] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
staggered
[ 25.928821] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
[ 25.928866] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
[ 25.928883] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13 value 2
[ 25.929020] ath10k_pci 0000:01:00.0: mac monitor recalc started? 1
needed? 1 allowed? 1
[ 25.941886] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
staggered
[ 25.941911] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
[ 25.941955] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
[ 25.941972] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13 value 2
[ 25.953727] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
staggered
[ 25.953753] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
[ 25.953798] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
[ 25.953817] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13 value 2
[ 25.970588] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
staggered
[ 25.970614] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
[ 25.970659] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
[ 25.970676] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13 value 2
[ 25.989056] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
staggered
[ 25.989081] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
[ 25.989126] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
[ 25.989143] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13 value 2
[ 26.071686] ath10k_pci 0000:01:00.0: mac vdev 0 peer create
18:a6:f7:26:0f:21 (new sta) sta 1 / 128 peer 2 / 144
[ 26.071712] ath10k_pci 0000:01:00.0: wmi peer create vdev_id 0
peer_addr 18:a6:f7:26:0f:21
[ 26.071952] ath10k_pci 0000:01:00.0: mac sta 18:a6:f7:26:0f:21 associated
[ 26.071981] ath10k_pci 0000:01:00.0: mac ht peer 18:a6:f7:26:0f:21
mcs cnt 24 nss 3
[ 26.071999] ath10k_pci 0000:01:00.0: mac vht peer 18:a6:f7:26:0f:21
max_mpdu 1048575 flags 0x601b001
[ 26.072013] ath10k_pci 0000:01:00.0: mac peer 18:a6:f7:26:0f:21 qos 1
[ 26.072028] ath10k_pci 0000:01:00.0: mac peer 18:a6:f7:26:0f:21
phymode 11ac-vht80
[ 26.072047] ath10k_pci 0000:01:00.0: wmi peer assoc vdev 0 addr
18:a6:f7:26:0f:21 (new)
[ 26.072071] ath10k_pci 0000:01:00.0: wmi vdev 0 peer
0x18:a6:f7:26:0f:21 set param 1 value 0
[ 26.072502] ath10k_pci 0000:01:00.0: received event id 36891 not
implemented
[ 26.074194] ath10k_pci 0000:01:00.0: mac sta rc update for
18:a6:f7:26:0f:21 changed 00000000 bw 2 nss 3 smps 1
[ 26.074586] ath10k_pci 0000:01:00.0: vdev 0 set beacon tx mode to
staggered
[ 26.074609] ath10k_pci 0000:01:00.0: wmi pdev set param 7 value 0
[ 26.074682] ath10k_pci 0000:01:00.0: mac vdev 0 dtim_period 2
[ 26.074701] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 13 value 2
[ 26.074760] ath10k_pci 0000:01:00.0: mac vdev 0 slot_time 2
[ 26.074779] ath10k_pci 0000:01:00.0: wmi vdev id 0x0 set param 7 value 2
[ 26.273652] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 26.730650] ath10k_pci 0000:01:00.0: wmi event debug mesg len 44
[ 26.773733] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID
[ 27.135445] ath10k_pci 0000:01:00.0: mac monitor recalc started? 1
needed? 1 allowed? 1
[ 27.273810] ath10k_pci 0000:01:00.0: WMI_UPDATE_STATS_EVENTID

It seems to start up in VHT80 mode and when it peers with Node B it
thinks Node B is also in VHT80 mode and ping works.

4. Beacons are sent at 6 Mb/s basic rate and unicast QoS Data is sent
with three spatial streams. Attached is the full pcap of the experiment.

Thank you for looking into this!

~Benjamin

On 12/05/2016 11:24 AM, Nagarajan, Ashok Raj wrote:
> 0x009A4577 0x00955B31
>
> Benjamin, Thanks for the logs.
> Quick questions to further debug the issue here,
>
> 1. Is this issue seen every time you start sending data traffic?
> 2. Issue seen with older firmwares? (FYR, http://linuxwireless.org/en/users/Drivers/ath10k/firmware/ )
> 3. Could you please share the dmesg from your device after enabling MAC and WMI logs in ath10k driver
> To enable debug logs please see http://linuxwireless.org/en/users/Drivers/ath10k/debug/
> 4. Do you know what is the Number of Spatial Streams seen in mesh beacons and in mesh data packet?
>
> Thanks,
> Ashok


Attachments:
ath10k_crash_pcap.tgz (24.50 kB)

2016-12-14 06:07:06

by Adrian Chadd

[permalink] [raw]
Subject: Re: ath10k firmware crashes in mesh mode on QCA9880

Hi!

Hm! So is there a firmware bug if there are 11n only capable nodes in
an 11s mesh?



-adrian

2016-12-14 05:43:16

by Alexis Green

[permalink] [raw]
Subject: Re: ath10k firmware crashes in mesh mode on QCA9880

Thank you for your help Rajkumar,

We've traced the problem down to a peering issue. Looks like there was
a missing compile flag that caused some kind of incongruence. My best
guest is that beacons are generated by firmware and advertise support
for AC mode, whereas wpa_supplicant, when not compiled with
CONFIG_IEEE80211AC=y, sends mesh peering messages and creates peers
without AC support, causing firmware to get confused. After
recompiling supplicant with the correct flag, no more crashes were
observed in casual testing. I submitted a pull request to LEDE to,
hopefully, fix it in upstream.

Best regards,

Alexis

On Tue, Dec 13, 2016 at 3:51 PM, Manoharan, Rajkumar
<[email protected]> wrote:
>> Tested the 10.2.4.70.59-2 firmware and wpa_supplicant running WITHOUT
>> encryption and it still crashes. I suspect this means wpa_supplicant is setting up
>> the interface incorrectly and/or transmitting a malformed packet that is causing
>> the driver to crash.
>>
> Ben,
>
> IIRC mesh support was validated in qca988x in VHT mode while ago. Either it could
> be regression in driver/fw or lede mac80211 package.
>
> 1) Could you please try plain backports in lede w/o applying ath10k patches.
> I do see 160MHz support in LEDE.
> 2) There are some peer stats dump from your earlier log. Disable peer stats
> by "peer_stats" debugfs.
> 3) Please confirm the behavior with older firmware revisions.
> 4) use iw to bring up open mesh to rule out wpa_s config
>
> -Rajkumar
>

2016-12-03 12:44:13

by Mohammed Shafi Shajakhan

[permalink] [raw]
Subject: Re: ath10k firmware crashes in mesh mode on QCA9880

Hi Benjamin,

On Fri, Dec 02, 2016 at 05:28:02PM -0800, Benjamin Morgan wrote:
> Just tried 10.2.4.70.58 firmware that you linked to and it still crashes:
>
> [ 131.568989] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
> 1838347e-9380-4a26-ac9d-2963ee95968b)
> [ 131.578124] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> 0x4100016c chip_id 0x043202ff sub 0000:0000
> [ 131.587491] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
> tracing 0 dfs 1 testmode 1
> [ 131.600521] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.58
> api 5 features no-p2p,raw-mode,mfp crc32 e1af076f
> [ 131.610899] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> crc32 bebc7c08
> [ 131.618325] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
> 2 cal file max-sta 128 raw 0 hwcrypto 1
> [ 131.629965] ath10k_pci 0000:01:00.0: firmware register dump:
> [ 131.635728] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
> 0x009A45AF 0x00955B31
> [ 131.643761] ath10k_pci 0000:01:00.0: [04]: 0x009A45AF 0x00060130
> 0x00000002 0x00439E98
> [ 131.651806] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
> 0x00407120 0x004436CC
> [ 131.659852] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
> 0x009A3550 0x009A355E
> [ 131.667892] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x009A31D6
> 0x00000000 0x00000000
> [ 131.675936] ath10k_pci 0000:01:00.0: [20]: 0x409A45AF 0x0040AAC4
> 0x0040AC60 0x0040AC09
> [ 131.683968] ath10k_pci 0000:01:00.0: [24]: 0x809A44F2 0x0040AB24
> 0x00400000 0xC09A45AF
> [ 131.692013] ath10k_pci 0000:01:00.0: [28]: 0x809A3A16 0x0040AB84
> 0x0044110C 0x00442074
> [ 131.700056] ath10k_pci 0000:01:00.0: [32]: 0x809A601A 0x0040ABB4
> 0x0044110C 0x00407120
> [ 131.708100] ath10k_pci 0000:01:00.0: [36]: 0x809A2EA4 0x0040ABF4
> 0x0040AC14 0x00001580
> [ 131.716143] ath10k_pci 0000:01:00.0: [40]: 0x80990F63 0x0040AD04
> 0x009C6458 0x004436CC
> [ 131.724175] ath10k_pci 0000:01:00.0: [44]: 0x80998520 0x0040AD64
> 0x004208FC 0x00439E4C
> [ 131.732220] ath10k_pci 0000:01:00.0: [48]: 0x8099AEA5 0x0040AD84
> 0x004208FC 0x00425874
> [ 131.740263] ath10k_pci 0000:01:00.0: [52]: 0x809BFC39 0x0040AEE4
> 0x00424FE8 0x00000002
> [ 131.748306] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
> 0x00000004 0x004039D0
> [ 131.857076] ieee80211 phy0: Hardware restart was requested
> [ 131.862705] ath10k_pci 0000:01:00.0: failed to synchronize
> monitor vdev 1 stop: -143
> [ 131.870594] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143

[shafi] request to try with the latest backports and ath.git
Also worth trying this fix in ath10k (once the next firmware version)
is available. I will keep you posted. thank you !

>
> ~Benjamin
>
> On 11/30/2016 11:09 PM, Mohammed Shafi Shajakhan wrote:
> >Hi,
> >
> >On Tue, Nov 29, 2016 at 11:22:12AM -0800, Benjamin Morgan wrote:
> >>When we try to transmit traffic (ping) between two meshed ath10k
> >>devices running latest lede we keep experiencing ath10k firmware
> >>crashes. This seems to only happen when running in 802.11n/ac mode
> >>but not in 802.11a/g mode. Also, from the station dumps it appears
> >>that management traffic is flowing between the devices, however when
> >>we try to send unicast data traffic the firmware crashes
> >>immediately.
> >[shafi] Did you get a chance to try with the below firmware as well
> >https://github.com/kvalo/ath10k-firmware/blob/master/QCA988X/hw2.0/10.2.4.70/firmware-5.bin_10.2.4.70.58
> >
> >>Platform: Archer C7 AC1750 v2
> >>Software Image: LEDE (HEAD, r2299) Commit: https://github.com/lede-project/source/commit/d596c21ebd5a3e6ce933eff3e51989031e4b1d58
> >>
> >>Crypto: wpa_supplicant
> >>wpa_supplicant-wlan0.conf
> >>network={
> >>ssid="bmorgan_lede_mesh"
> >>key_mgmt=SAE
> >>mode=5
> >>frequency=5180
> >>psk="meshpassword"
> >>}
> >>
> >>Backports Verstion:
> >>[ 9.818007] Loading modules backported from Linux version
> >>wt-2016-10-03-1-g6fcb1a6
> >>[ 9.825736] Backport generated by backports.git
> >>backports-20160324-9-g0e38f5c
> >>
> >>​​Ath10k Initialization on Station A (dmesg)
> >>[ 9.896715] PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
> >>[ 9.902622] ath10k_pci 0000:01:00.0: pci irq legacy oper_irq_mode
> >>1 irq_mode 0 reset_mode 0
> >>[ 10.123734] ath10k_pci 0000:01:00.0: Direct firmware load for
> >>ath10k/pre-cal-pci-0000:01:00.0.bin failed with error -2
> >>[ 10.134620] ath10k_pci 0000:01:00.0: Falling back to user helper
> >>[ 10.287680] firmware ath10k!pre-cal-pci-0000:01:00.0.bin:
> >>firmware_loading_store: map pages failed
> >>[ 10.622789] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> >>0x4100016c chip_id 0x043202ff sub 0000:0000
> >>[ 10.632184] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
> >>tracing 0 dfs 1 testmode 1
> >>[ 10.645231] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
> >>api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
> >>[ 10.655660] ath10k_pci 0000:01:00.0: Direct firmware load for
> >>ath10k/QCA988X/hw2.0/board-2.bin failed with error -2
> >>[ 10.666264] ath10k_pci 0000:01:00.0: Falling back to user helper
> >>[ 10.747925] firmware ath10k!QCA988X!hw2.0!board-2.bin:
> >>firmware_loading_store: map pages failed
> >>[ 11.011123] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> >>crc32 bebc7c08
> >>[ 12.155224] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
> >>2 cal file max-sta 128 raw 0 hwcrypto 1
> >>
> >>Station A (wlan0):
> >>18:A6:F7:23:6E:66
> >>10.230.5.41
> >>
> >>Station B (wlan0):
> >>18:a6:f7:26:0f:21
> >>10.230.5.42
> >>
> >>Station Dump on Station A before ping:
> >>Station 18:a6:f7:26:0f:21 (on wlan0)
> >>inactive time:340 ms
> >>rx bytes:2472
> >>rx packets:28
> >>tx bytes:1204
> >>tx packets:9
> >>tx retries:0
> >>tx failed:0
> >>rx drop misc:1
> >>signal: -14 dBm
> >>signal avg:-14 dBm
> >>Toffset:18142530 us
> >>tx bitrate:6.0 MBit/s
> >>rx bitrate:6.0 MBit/s
> >>rx duration:1524 us
> >>mesh llid:0
> >>mesh plid:0
> >>mesh plink:ESTAB
> >>mesh local PS mode:ACTIVE
> >>mesh peer PS mode:UNKNOWN
> >>mesh non-peer PS mode:ACTIVE
> >>authorized:yes
> >>authenticated:yes
> >>associated:yes
> >>preamble:long
> >>WMM/WME:yes
> >>MFP:yes
> >>TDLS peer:no
> >>DTIM period:2
> >>beacon interval:1000
> >>connected time:10 seconds
> >>
> >>​Crash Log on Station B (10.230.5.42)
> >>[245.483888] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
> >>2bab5ee9-08ff-4a17-95b1-636d212acebc)
> >>[245.493020] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> >>0x4100016c chip_id 0x043202ff sub 0000:0000
> >>[245.502384] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
> >>tracing 0 dfs 1 testmode 1
> >>[245.515436] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api
> >>5 features no-p2p,raw-mode,mfp crc32 9d340dd9
> >>[245.525812] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> >>crc32 bebc7c08
> >>[245.533232] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
> >>cal file max-sta 128 raw 0 hwcrypto 1
> >>[245.544876] ath10k_pci 0000:01:00.0: firmware register dump:
> >>[245.550633] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
> >>0x009A4577 0x00955B31
> >>[245.558676] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
> >>0x00000002 0x00439E98
> >>[245.566715] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
> >>0x00407120 0x004436CC
> >>[245.574749] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
> >>0x009A3518 0x009A3526
> >>[245.582793] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
> >>0x00000000 0x00000000
> >>[245.590836] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
> >>0x0040AC60 0x0040AC09
> >>[245.598882] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
> >>0x00400000 0xC09A4577
> >>[245.606923] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
> >>0x0044110C 0x00442074
> >>[245.614955] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
> >>0x0044110C 0x00407120
> >>[245.623000] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
> >>0x0040AC14 0x00001580
> >>[245.631043] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
> >>0x009C643C 0x004436CC
> >>[245.639086] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
> >>0x004208FC 0x00439E4C
> >>[245.647129] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
> >>0x004208FC 0x00425404
> >>[245.655170] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
> >>0x00424FE8 0x00000002
> >>[245.663198] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
> >>0x00000004 0x004039D0
> >>[245.767023] ieee80211 phy0: Hardware restart was requested
> >>[245.772655] ath10k_pci 0000:01:00.0: failed to synchronize monitor
> >>vdev 1 stop: -143
> >>[245.780542] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
> >>​
> >>With wpa_supplicant turned off:
> >>
> >>Station Dump on Station A before ping:
> >>Station 18:a6:f7:26:0f:21 (on wlan0)
> >>inactive time:60 ms
> >>rx bytes:3874
> >>rx packets:67
> >>tx bytes:707
> >>tx packets:7
> >>tx retries:0
> >>tx failed:1
> >>rx drop misc:3
> >>signal: -14 dBm
> >>signal avg:-14 dBm
> >>Toffset:18446744073684250999 us
> >>tx bitrate:6.0 MBit/s
> >>rx bitrate:6.0 MBit/s
> >>rx duration:1312 us
> >>mesh llid:24953
> >>mesh plid:59412
> >>mesh plink:ESTAB
> >>mesh local PS mode:ACTIVE
> >>mesh peer PS mode:UNKNOWN
> >>mesh non-peer PS mode:ACTIVE
> >>authorized:yes
> >>authenticated:yes
> >>associated:yes
> >>preamble:long
> >>WMM/WME:yes
> >>MFP:no
> >>TDLS peer:no
> >>DTIM period:2
> >>beacon interval:1000
> >>short slot time:yes
> >>connected time:33 seconds
> >>
> >>Firmware crash on Station A (10.230.5.41)
> >>[ 1040.984599] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
> >>42d4f6dc-0e89-4505-aefb-58cbb70d0dce)
> >>[ 1040.993732] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> >>0x4100016c chip_id 0x043202ff sub 0000:0000
> >>[ 1041.003100] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
> >>tracing 0 dfs 1 testmode 1
> >>[ 1041.016144] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
> >>api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
> >>[ 1041.026523] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> >>crc32 bebc7c08
> >>[ 1041.033940] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
> >>2 cal file max-sta 128 raw 0 hwcrypto 1
> >>[ 1041.045593] ath10k_pci 0000:01:00.0: firmware register dump:
> >>[ 1041.051336] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
> >>0x009A4577 0x00955B31
> >>[ 1041.059383] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
> >>0x00000001 0x00435354
> >>[ 1041.067424] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
> >>0x00407120 0x004436CC
> >>[ 1041.075463] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
> >>0x009A43A0 0x009A43DE
> >>[ 1041.083490] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
> >>0x00000000 0x00000000
> >>[ 1041.091535] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
> >>0x0040AC60 0x0040AC09
> >>[ 1041.099579] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
> >>0x00955A00 0xC09A4577
> >>[ 1041.107622] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
> >>0x0044110C 0x00442074
> >>[ 1041.115661] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
> >>0x0044110C 0x00407120
> >>[ 1041.123689] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
> >>0x0040AC10 0x00001580
> >>[ 1041.131733] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
> >>0x009C643C 0x004436CC
> >>[ 1041.139777] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
> >>0x004208FC 0x00439E4C
> >>[ 1041.147820] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
> >>0x004208FC 0x00425758
> >>[ 1041.155860] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
> >>0x00424FE8 0x00000002
> >>[ 1041.163888] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
> >>0x00000004 0x004039D0
> >>[ 1041.267025] ieee80211 phy0: Hardware restart was requested
> >>[ 1041.272656] ath10k_pci 0000:01:00.0: failed to synchronize
> >>monitor vdev 1 stop: -143
> >>[ 1041.280545] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
> >>
> >>​
> >>Thanks in advance for your time and help,
> >>
> >>~Benjamin
> >>
>

2016-12-14 06:06:29

by Alexis Green

[permalink] [raw]
Subject: Re: ath10k firmware crashes in mesh mode on QCA9880

Hi Adrian,

I have not done much testing of ath10k and ath9k devices in a single
encrypted mesh recently, but I have a memory of only having this issue
when communicating between ath10k devices.

Alexis

On Tue, Dec 13, 2016 at 9:53 PM, Adrian Chadd <[email protected]> wrote:
> Hi!
>
> Hm! So is there a firmware bug if there are 11n only capable nodes in
> an 11s mesh?
>
>
>
> -adrian

2016-12-01 07:09:18

by Mohammed Shafi Shajakhan

[permalink] [raw]
Subject: Re: ath10k firmware crashes in mesh mode on QCA9880

Hi,

On Tue, Nov 29, 2016 at 11:22:12AM -0800, Benjamin Morgan wrote:
> When we try to transmit traffic (ping) between two meshed ath10k
> devices running latest lede we keep experiencing ath10k firmware
> crashes. This seems to only happen when running in 802.11n/ac mode
> but not in 802.11a/g mode. Also, from the station dumps it appears
> that management traffic is flowing between the devices, however when
> we try to send unicast data traffic the firmware crashes
> immediately.

[shafi] Did you get a chance to try with the below firmware as well
https://github.com/kvalo/ath10k-firmware/blob/master/QCA988X/hw2.0/10.2.4.70/firmware-5.bin_10.2.4.70.58

>
> Platform: Archer C7 AC1750 v2
> Software Image: LEDE (HEAD, r2299) Commit: https://github.com/lede-project/source/commit/d596c21ebd5a3e6ce933eff3e51989031e4b1d58
>
> Crypto: wpa_supplicant
> wpa_supplicant-wlan0.conf
> network={
> ssid="bmorgan_lede_mesh"
> key_mgmt=SAE
> mode=5
> frequency=5180
> psk="meshpassword"
> }
>
> Backports Verstion:
> [ 9.818007] Loading modules backported from Linux version
> wt-2016-10-03-1-g6fcb1a6
> [ 9.825736] Backport generated by backports.git
> backports-20160324-9-g0e38f5c
>
> ​​Ath10k Initialization on Station A (dmesg)
> [ 9.896715] PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
> [ 9.902622] ath10k_pci 0000:01:00.0: pci irq legacy oper_irq_mode
> 1 irq_mode 0 reset_mode 0
> [ 10.123734] ath10k_pci 0000:01:00.0: Direct firmware load for
> ath10k/pre-cal-pci-0000:01:00.0.bin failed with error -2
> [ 10.134620] ath10k_pci 0000:01:00.0: Falling back to user helper
> [ 10.287680] firmware ath10k!pre-cal-pci-0000:01:00.0.bin:
> firmware_loading_store: map pages failed
> [ 10.622789] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> 0x4100016c chip_id 0x043202ff sub 0000:0000
> [ 10.632184] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
> tracing 0 dfs 1 testmode 1
> [ 10.645231] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
> api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
> [ 10.655660] ath10k_pci 0000:01:00.0: Direct firmware load for
> ath10k/QCA988X/hw2.0/board-2.bin failed with error -2
> [ 10.666264] ath10k_pci 0000:01:00.0: Falling back to user helper
> [ 10.747925] firmware ath10k!QCA988X!hw2.0!board-2.bin:
> firmware_loading_store: map pages failed
> [ 11.011123] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> crc32 bebc7c08
> [ 12.155224] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
> 2 cal file max-sta 128 raw 0 hwcrypto 1
>
> Station A (wlan0):
> 18:A6:F7:23:6E:66
> 10.230.5.41
>
> Station B (wlan0):
> 18:a6:f7:26:0f:21
> 10.230.5.42
>
> Station Dump on Station A before ping:
> Station 18:a6:f7:26:0f:21 (on wlan0)
> inactive time:340 ms
> rx bytes:2472
> rx packets:28
> tx bytes:1204
> tx packets:9
> tx retries:0
> tx failed:0
> rx drop misc:1
> signal: -14 dBm
> signal avg:-14 dBm
> Toffset:18142530 us
> tx bitrate:6.0 MBit/s
> rx bitrate:6.0 MBit/s
> rx duration:1524 us
> mesh llid:0
> mesh plid:0
> mesh plink:ESTAB
> mesh local PS mode:ACTIVE
> mesh peer PS mode:UNKNOWN
> mesh non-peer PS mode:ACTIVE
> authorized:yes
> authenticated:yes
> associated:yes
> preamble:long
> WMM/WME:yes
> MFP:yes
> TDLS peer:no
> DTIM period:2
> beacon interval:1000
> connected time:10 seconds
>
> ​Crash Log on Station B (10.230.5.42)
> [245.483888] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
> 2bab5ee9-08ff-4a17-95b1-636d212acebc)
> [245.493020] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> 0x4100016c chip_id 0x043202ff sub 0000:0000
> [245.502384] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
> tracing 0 dfs 1 testmode 1
> [245.515436] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api
> 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
> [245.525812] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> crc32 bebc7c08
> [245.533232] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
> cal file max-sta 128 raw 0 hwcrypto 1
> [245.544876] ath10k_pci 0000:01:00.0: firmware register dump:
> [245.550633] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
> 0x009A4577 0x00955B31
> [245.558676] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
> 0x00000002 0x00439E98
> [245.566715] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
> 0x00407120 0x004436CC
> [245.574749] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
> 0x009A3518 0x009A3526
> [245.582793] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
> 0x00000000 0x00000000
> [245.590836] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
> 0x0040AC60 0x0040AC09
> [245.598882] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
> 0x00400000 0xC09A4577
> [245.606923] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
> 0x0044110C 0x00442074
> [245.614955] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
> 0x0044110C 0x00407120
> [245.623000] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
> 0x0040AC14 0x00001580
> [245.631043] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
> 0x009C643C 0x004436CC
> [245.639086] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
> 0x004208FC 0x00439E4C
> [245.647129] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
> 0x004208FC 0x00425404
> [245.655170] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
> 0x00424FE8 0x00000002
> [245.663198] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
> 0x00000004 0x004039D0
> [245.767023] ieee80211 phy0: Hardware restart was requested
> [245.772655] ath10k_pci 0000:01:00.0: failed to synchronize monitor
> vdev 1 stop: -143
> [245.780542] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
> ​
> With wpa_supplicant turned off:
>
> Station Dump on Station A before ping:
> Station 18:a6:f7:26:0f:21 (on wlan0)
> inactive time:60 ms
> rx bytes:3874
> rx packets:67
> tx bytes:707
> tx packets:7
> tx retries:0
> tx failed:1
> rx drop misc:3
> signal: -14 dBm
> signal avg:-14 dBm
> Toffset:18446744073684250999 us
> tx bitrate:6.0 MBit/s
> rx bitrate:6.0 MBit/s
> rx duration:1312 us
> mesh llid:24953
> mesh plid:59412
> mesh plink:ESTAB
> mesh local PS mode:ACTIVE
> mesh peer PS mode:UNKNOWN
> mesh non-peer PS mode:ACTIVE
> authorized:yes
> authenticated:yes
> associated:yes
> preamble:long
> WMM/WME:yes
> MFP:no
> TDLS peer:no
> DTIM period:2
> beacon interval:1000
> short slot time:yes
> connected time:33 seconds
>
> Firmware crash on Station A (10.230.5.41)
> [ 1040.984599] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
> 42d4f6dc-0e89-4505-aefb-58cbb70d0dce)
> [ 1040.993732] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> 0x4100016c chip_id 0x043202ff sub 0000:0000
> [ 1041.003100] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
> tracing 0 dfs 1 testmode 1
> [ 1041.016144] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
> api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
> [ 1041.026523] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> crc32 bebc7c08
> [ 1041.033940] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
> 2 cal file max-sta 128 raw 0 hwcrypto 1
> [ 1041.045593] ath10k_pci 0000:01:00.0: firmware register dump:
> [ 1041.051336] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
> 0x009A4577 0x00955B31
> [ 1041.059383] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
> 0x00000001 0x00435354
> [ 1041.067424] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
> 0x00407120 0x004436CC
> [ 1041.075463] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
> 0x009A43A0 0x009A43DE
> [ 1041.083490] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
> 0x00000000 0x00000000
> [ 1041.091535] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
> 0x0040AC60 0x0040AC09
> [ 1041.099579] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
> 0x00955A00 0xC09A4577
> [ 1041.107622] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
> 0x0044110C 0x00442074
> [ 1041.115661] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
> 0x0044110C 0x00407120
> [ 1041.123689] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
> 0x0040AC10 0x00001580
> [ 1041.131733] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
> 0x009C643C 0x004436CC
> [ 1041.139777] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
> 0x004208FC 0x00439E4C
> [ 1041.147820] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
> 0x004208FC 0x00425758
> [ 1041.155860] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
> 0x00424FE8 0x00000002
> [ 1041.163888] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
> 0x00000004 0x004039D0
> [ 1041.267025] ieee80211 phy0: Hardware restart was requested
> [ 1041.272656] ath10k_pci 0000:01:00.0: failed to synchronize
> monitor vdev 1 stop: -143
> [ 1041.280545] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
>
> ​
> Thanks in advance for your time and help,
>
> ~Benjamin
>