2018-10-24 05:11:41

by Ellie Reeves

[permalink] [raw]
Subject: ath9k driver may be broken on ARM64 ?

Hi,
for the context part:

I've got a marvell espressobin board, to which I attached a mini-PCIE
wireless card, atheros AR9565 aka. QCB335. I am using both archlinux-arm
and a custom OS made with buildroot, and both have this problem.

Problem:

When the system boots up, the kernel recognizes the card, and then the
ath9k driver gets loaded. From what someone on the irc channel of
#linux-wireless determined, the driver ends up doing a sigbus while
attempting to read from a specific address in memory, which of course
makes it crash. We did check the page tables, and the memory seems to be
mapped fine. So we're still clueless as to why it is not possible to
read from that particular memory address, and why the driver would do
so. Here I attach the kernel page tables, along with the dmesg output.
The problematic line in the trace is x19.

I'd appreciate any help in fixing this issue which I think might be
related to ARM64.

Note that I am no programmer, all the info I gathered so far were with
someone else's help. But if you need some more debug output, files or
anything else, just let me know. I can also apply patches to test
things, or modify the code if I know exactly in what way, i.e: what to
copy and past.

Dmesg:

http://ix.io/1pTK

kernel page tables -- from a later kernel build, but do notice that the
memory is still mapped correctly even if the register changed:

http://ix.io/1pQh

Thanks a lot :)


2018-10-24 12:09:55

by Tom Psyborg

[permalink] [raw]
Subject: Re: ath9k driver may be broken on ARM64 ?

Hi

There was similar issue reported in forum recently:
https://forum.openwrt.org/t/wireless-card-atheros-ar9565/23608

Could you enable ath9k debug and post output?

2018-10-24 18:37:34

by Ellie Reeves

[permalink] [raw]
Subject: Re: ath9k driver may be broken on ARM64 ?

Hi,
sure, I enabled atheros wireless debugging in my kernel and am
rebuilding it. After this is done, should I set ath9k.debug= on the
cmdline ? It is built as module, but my filesystem is read-only so no
modprobe.d snippet can be added.
Thanks

Tom Psyborg a écrit :
> Hi
>
> There was similar issue reported in forum recently:
> https://forum.openwrt.org/t/wireless-card-atheros-ar9565/23608
>
> Could you enable ath9k debug and post output?


2018-10-24 19:13:35

by Ellie Reeves

[permalink] [raw]
Subject: Re: ath9k driver may be broken on ARM64 ?

Hi,
well I enabled CONFIG_ATH_DEBUG and booted this kernel, sorry for the
mess of escape sequence but I captured via serial console:
http://ix.io/1pWI
I don't see anything different even if I cranked up ath9k.debug= to the
*any* value. I guess this thing happens so early at boot that nothing
useful is yet initialized...
Thanks

Ellie Revves a écrit :
> Hi,
> sure, I enabled atheros wireless debugging in my kernel and am
> rebuilding it. After this is done, should I set ath9k.debug= on the
> cmdline ? It is built as module, but my filesystem is read-only so no
> modprobe.d snippet can be added.
> Thanks
>
> Tom Psyborg a écrit :
>> Hi
>>
>> There was similar issue reported in forum recently:
>> https://forum.openwrt.org/t/wireless-card-atheros-ar9565/23608
>>
>> Could you enable ath9k debug and post output?
>


2018-10-24 22:54:00

by Tom Psyborg

[permalink] [raw]
Subject: Re: ath9k driver may be broken on ARM64 ?

Hi

Try selecting config option ATH9K_SUPPORT_PCOEM (Support chips used in
PC OEM cards) and rebuild your image.

I'm not familiar with your architecture but it seems you're not
getting debug output. It should print a lot of info right after the:
ath: phy0: WB335 2-ANT card detected

2018-10-24 23:03:36

by Ellie Reeves

[permalink] [raw]
Subject: Re: ath9k driver may be broken on ARM64 ?

Hi,
CONFIG_ATH9K_PCOEM is already enabled in my kernel build. In fact if it
is disabled my card does not show up at all.
Thanks

Tom Psyborg a écrit :
> Hi
>
> Try selecting config option ATH9K_SUPPORT_PCOEM (Support chips used in
> PC OEM cards) and rebuild your image.
>
> I'm not familiar with your architecture but it seems you're not
> getting debug output. It should print a lot of info right after the:
> ath: phy0: WB335 2-ANT card detected


2018-10-24 23:18:29

by Tom Psyborg

[permalink] [raw]
Subject: Re: ath9k driver may be broken on ARM64 ?

if your system is read-only then you should make a build with debug
option enabled:

inside your buildroot create this path: files/etc/modules.d/ and put
file named ath9k there which should contain "ath9k debug=0xffffffff"
(more info: https://openwrt.org/docs/guide-developer/build-system/use-buildsystem#custom_files)

rebuild and try boot again capturing bootlog

2018-10-25 04:24:45

by Ellie Reeves

[permalink] [raw]
Subject: Re: ath9k driver may be broken on ARM64 ?

Here's another boot log along with some commands after the system booted
up to show what I did. I don't think I did anything wrong, and yet this
is no different, debug option appears to be completely useless.
http://ix.io/1pY0
Thanks

Tom Psyborg a écrit :
> if your system is read-only then you should make a build with debug
> option enabled:
>
> inside your buildroot create this path: files/etc/modules.d/ and put
> file named ath9k there which should contain "ath9k debug=0xffffffff"
> (more info: https://openwrt.org/docs/guide-developer/build-system/use-buildsystem#custom_files)
>
> rebuild and try boot again capturing bootlog


2018-10-25 06:33:24

by Dominique Martinet

[permalink] [raw]
Subject: Re: ath9k driver may be broken on ARM64 ?

Ellie Revves wrote on Thu, Oct 25, 2018:
> Here's another boot log along with some commands after the system
> booted up to show what I did. I don't think I did anything wrong,
> and yet this is no different, debug option appears to be completely
> useless.

I'm not familiar with ath at all, just happened to be free at the time
this was brought up on IRC (#linux-wireless), so can relate what we said
there after looking at the debug symbols to interprete the stack a bit
further. It seems to fail before any print, honestly.

In the stack trace, x17 (0x4020) is reg_offset, so looking from the
value I think it would be AR_INTR_SYNC_CAUSE which should be in some
path like this inlined in ath9k_hw_init -> __ath9k_hw_init ->
ath9k_hw_set_reset_reg -> ath9k_hw_set_reset_power_on ->
ath9k_hw_set_reset -> tmpReg = REG_READ(ah, AR_INTR_SYNC_CAUSE);


The only thing that would have happened before is the ath_pci_probe call
that sets up the memory but even that doesn't log anything.

What I find surprising is that the mapping for sc->mem is done (as shown
in the kernel page table dump), and some REG_WRITE worked before, but
the first REG_READ fails despite being an aligned address so something
is odd with how the memory is setup?


I think it would be interesting to add some debug statement early in
init (pci init to check how the memory is setup, if I'm not wrong about
it being pci, and along the path I described earlier) ; but my free time
ended long ago so don't think I'll be of much more use from now :p

--
Dominique Martinet | Asmadeus

2018-10-26 05:12:13

by Dominique Martinet

[permalink] [raw]
Subject: Re: ath9k driver may be broken on ARM64 ?

Dominique Martinet wrote on Thu, Oct 25, 2018:
> What I find surprising is that the mapping for sc->mem is done (as shown
> in the kernel page table dump), and some REG_WRITE worked before, but
> the first REG_READ fails despite being an aligned address so something
> is odd with how the memory is setup?
>
>
> I think it would be interesting to add some debug statement early in
> init (pci init to check how the memory is setup, if I'm not wrong about
> it being pci, and along the path I described earlier) ; but my free time
> ended long ago so don't think I'll be of much more use from now :p

(not sure if you got notified off list, but fwiw the bug is apparently
resolved by upgrading to a newer release of u-boot - same version, from
u-boot 2017.03 armada v17.06 to u-boot 2017.03 armada v17.10
I have no idea what they're doing in u-boot with the memory setup but I
guess the case is closed...)

--
Asmadeus | Dominique Martinet