LinuxLists.cc - AR9330 hornet board stops beaconing after a few days (0xdeadbeef)

2012-08-13 17:24:25

Subject: AR9330 hornet board stops beaconing after a few days (0xdeadbeef)

Hello ath9k fellows,

I'm using a pretty new AR9330 rev1/hornet based AP at home. After some
days (5 days this time), it suddenly stops beaconing. dmesg shows nothing,
hostapd is still running, but what I can see is:

cat /sys/kernel/debug/ieee80211/phy0/ath9k/regdump
0x000000 0xdeadbeef
0x000004 0xdeadbeef
0x000008 0xdeadbeef
0x00000c 0xdeadbeef
0x000010 0xdeadbeef
0x000014 0xdeadbeef
[...]
0x001ff4 0xdeadbeef
0x001ff8 0xdeadbeef
0x001ffc 0xdeadbeef
0x002000 0xbadc0ffe
0x002004 0xbadc0ffe
0x002008 0xbadc0ffe
0x00200c 0xbadc0ffe
0x002010 0xbadc0ffe
[...]
0x003ff0 0xbadc0ffe
0x003ff4 0xbadc0ffe
0x003ff8 0xbadc0ffe
0x003ffc 0xbadc0ffe
0x004000 0x00000000
0x004004 0x0102420b
0x004008 0x00000000
0x00400c 0x00000000
[...] (some more sane looking registers here, and then zeros)
0x004ef8 0x00000000
0x004efc 0x00000000
0x004f00 0xdeadbeef
0x004f04 0xdeadbeef
0x004f08 0xdeadbeef
[...] (and the rest is mostly deadbeef)

Also /sys/kernel/debug/ieee80211/phy0/ath9k/dma shows a lot of deadbeef.

I'm using OpenWRT r31729, a single AP bridged with Ethernet. It is normal
home usage, i.e. a few laptops and android phones doing not so much traffic.
The box stops working out of the blue after a few days without any outside
event (at least none I'm aware of). Sometimes this problem does not occur for
2 weeks, sometimes after 12 hours, I couldn't find an easy way to reproduce it
yet. As said, the box is perfectly reachable via SSH, no errors in syslog or
anywhere else. Also an ifconfig wlan0 down and up seems to fix the problem.

I'll try a new firmware with more debugging turned on, but have any of you
seen this problem? Any input would be very much appreciated. I have a complete
/sys/kernel/debug log from a working and a non-working system, along with
more info (brctl, ip config, etc) which I'd love to share with anyone interested
- it's too much to post on a public ml.

Thank you very much
Cheers,
Simon

Attachments:

(No filename) (1.88 kB)
signature.asc (198.00 B)
Digital signature Download all attachments

2012-08-23 14:59:48

by Mohammed Shafi

[permalink] [raw]

Subject: Re: AR9330 hornet board stops beaconing after a few days (0xdeadbeef)

On Wed, Aug 22, 2012 at 10:50 AM, Mohammed Shafi
<[email protected]> wrote:
> Hi Simon,
>
> On Mon, Aug 20, 2012 at 3:34 AM, Simon Wunderlich
> <[email protected]> wrote:
>> Hello,
>>
>> just to bump again - isn't there anyone who can give us some hint or
>> clue regarding these 0xdeadbeef entries in the registers? Couldn't find
>> a connection to other reports deadbeef on the mailing list so far.
>
> based on the suggestion from Felix Bitterli he says the deadbeef is
> due to clocks
> of MAC/baseband is off. Adrian says dumping RTC registers might
> provide some clue
>
> today evening i would just take a look at some h/w specific changes
> that were still yet to be in ath9k.

sorry, i am unable to figure out any commits that seems to fix the symptoms
like this. i will dig through or ask some one internally.
please check with the logs sudo modprobe -v ath9k debug=0x8601
with beacon stuck debug enabled

>
>
>>
>> Thanks!
>> Simon
>>
>> On Mon, Aug 13, 2012 at 06:53:40PM +0200, Simon Wunderlich wrote:
>>> Hello ath9k fellows,
>>>
>>> I'm using a pretty new AR9330 rev1/hornet based AP at home. After some
>>> days (5 days this time), it suddenly stops beaconing. dmesg shows nothing,
>>> hostapd is still running, but what I can see is:
>>>
>>> cat /sys/kernel/debug/ieee80211/phy0/ath9k/regdump
>>> 0x000000 0xdeadbeef
>>> 0x000004 0xdeadbeef
>>> 0x000008 0xdeadbeef
>>> 0x00000c 0xdeadbeef
>>> 0x000010 0xdeadbeef
>>> 0x000014 0xdeadbeef
>>> [...]
>>> 0x001ff4 0xdeadbeef
>>> 0x001ff8 0xdeadbeef
>>> 0x001ffc 0xdeadbeef
>>> 0x002000 0xbadc0ffe
>>> 0x002004 0xbadc0ffe
>>> 0x002008 0xbadc0ffe
>>> 0x00200c 0xbadc0ffe
>>> 0x002010 0xbadc0ffe
>>> [...]
>>> 0x003ff0 0xbadc0ffe
>>> 0x003ff4 0xbadc0ffe
>>> 0x003ff8 0xbadc0ffe
>>> 0x003ffc 0xbadc0ffe
>>> 0x004000 0x00000000
>>> 0x004004 0x0102420b
>>> 0x004008 0x00000000
>>> 0x00400c 0x00000000
>>> [...] (some more sane looking registers here, and then zeros)
>>> 0x004ef8 0x00000000
>>> 0x004efc 0x00000000
>>> 0x004f00 0xdeadbeef
>>> 0x004f04 0xdeadbeef
>>> 0x004f08 0xdeadbeef
>>> [...] (and the rest is mostly deadbeef)
>>>
>>> Also /sys/kernel/debug/ieee80211/phy0/ath9k/dma shows a lot of deadbeef.
>>>
>>> I'm using OpenWRT r31729, a single AP bridged with Ethernet. It is normal
>>> home usage, i.e. a few laptops and android phones doing not so much traffic.
>>> The box stops working out of the blue after a few days without any outside
>>> event (at least none I'm aware of). Sometimes this problem does not occur for
>>> 2 weeks, sometimes after 12 hours, I couldn't find an easy way to reproduce it
>>> yet. As said, the box is perfectly reachable via SSH, no errors in syslog or
>>> anywhere else. Also an ifconfig wlan0 down and up seems to fix the problem.
>>>
>>> I'll try a new firmware with more debugging turned on, but have any of you
>>> seen this problem? Any input would be very much appreciated. I have a complete
>>> /sys/kernel/debug log from a working and a non-working system, along with
>>> more info (brctl, ip config, etc) which I'd love to share with anyone interested
>>> - it's too much to post on a public ml.
>>>
>>> Thank you very much
>>> Cheers,
>>> Simon
>>
>>
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.10 (GNU/Linux)
>>
>> iEYEARECAAYFAlAxYvQACgkQrzg/fFk7axaPhACfaTuTkdyx9J/Q4z4EQBf/3sfi
>> O/oAn0c8YFa8FPzayVApD2qX78OBz8AT
>> =3WcZ
>> -----END PGP SIGNATURE-----
>>
>
>
>
> --
> thanks,
> shafi

--
thanks,
shafi

2012-08-22 05:22:05

by Mohammed Shafi

[permalink] [raw]

Subject: Re: AR9330 hornet board stops beaconing after a few days (0xdeadbeef)

Hi Simon,

On Mon, Aug 20, 2012 at 3:34 AM, Simon Wunderlich
<[email protected]> wrote:
> Hello,
>
> just to bump again - isn't there anyone who can give us some hint or
> clue regarding these 0xdeadbeef entries in the registers? Couldn't find
> a connection to other reports deadbeef on the mailing list so far.

based on the suggestion from Felix Bitterli he says the deadbeef is
due to clocks
of MAC/baseband is off. Adrian says dumping RTC registers might
provide some clue

today evening i would just take a look at some h/w specific changes
that were still yet to be in ath9k.

>
> Thanks!
> Simon
>
> On Mon, Aug 13, 2012 at 06:53:40PM +0200, Simon Wunderlich wrote:
>> Hello ath9k fellows,
>>
>> I'm using a pretty new AR9330 rev1/hornet based AP at home. After some
>> days (5 days this time), it suddenly stops beaconing. dmesg shows nothing,
>> hostapd is still running, but what I can see is:
>>
>> cat /sys/kernel/debug/ieee80211/phy0/ath9k/regdump
>> 0x000000 0xdeadbeef
>> 0x000004 0xdeadbeef
>> 0x000008 0xdeadbeef
>> 0x00000c 0xdeadbeef
>> 0x000010 0xdeadbeef
>> 0x000014 0xdeadbeef
>> [...]
>> 0x001ff4 0xdeadbeef
>> 0x001ff8 0xdeadbeef
>> 0x001ffc 0xdeadbeef
>> 0x002000 0xbadc0ffe
>> 0x002004 0xbadc0ffe
>> 0x002008 0xbadc0ffe
>> 0x00200c 0xbadc0ffe
>> 0x002010 0xbadc0ffe
>> [...]
>> 0x003ff0 0xbadc0ffe
>> 0x003ff4 0xbadc0ffe
>> 0x003ff8 0xbadc0ffe
>> 0x003ffc 0xbadc0ffe
>> 0x004000 0x00000000
>> 0x004004 0x0102420b
>> 0x004008 0x00000000
>> 0x00400c 0x00000000
>> [...] (some more sane looking registers here, and then zeros)
>> 0x004ef8 0x00000000
>> 0x004efc 0x00000000
>> 0x004f00 0xdeadbeef
>> 0x004f04 0xdeadbeef
>> 0x004f08 0xdeadbeef
>> [...] (and the rest is mostly deadbeef)
>>
>> Also /sys/kernel/debug/ieee80211/phy0/ath9k/dma shows a lot of deadbeef.
>>
>> I'm using OpenWRT r31729, a single AP bridged with Ethernet. It is normal
>> home usage, i.e. a few laptops and android phones doing not so much traffic.
>> The box stops working out of the blue after a few days without any outside
>> event (at least none I'm aware of). Sometimes this problem does not occur for
>> 2 weeks, sometimes after 12 hours, I couldn't find an easy way to reproduce it
>> yet. As said, the box is perfectly reachable via SSH, no errors in syslog or
>> anywhere else. Also an ifconfig wlan0 down and up seems to fix the problem.
>>
>> I'll try a new firmware with more debugging turned on, but have any of you
>> seen this problem? Any input would be very much appreciated. I have a complete
>> /sys/kernel/debug log from a working and a non-working system, along with
>> more info (brctl, ip config, etc) which I'd love to share with anyone interested
>> - it's too much to post on a public ml.
>>
>> Thank you very much
>> Cheers,
>> Simon
>
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iEYEARECAAYFAlAxYvQACgkQrzg/fFk7axaPhACfaTuTkdyx9J/Q4z4EQBf/3sfi
> O/oAn0c8YFa8FPzayVApD2qX78OBz8AT
> =3WcZ
> -----END PGP SIGNATURE-----
>

--
thanks,
shafi

2012-08-23 09:28:29

by Simon Wunderlich

[permalink] [raw]

Subject: Re: AR9330 hornet board stops beaconing after a few days (0xdeadbeef)

Hello Adrian,

thanks for your comments!

On Wed, Aug 22, 2012 at 11:57:04AM -0700, Adrian Chadd wrote:
> Yeah. The deadbeef means "something's turned off."
>
> I'd start with the SoC reset register and see if the MAC/WMAC bits are
> correctly set. Ie, that something hasn't gone and reset the wireless
> bits behind your back.

Sure, but which register would that be? How can I find out? Is it included
in regdump of ath9k debugfs?

>
> You could try hacking up a work around that does a full MAC/WMAC reset
> when you reset the driver.
>
> Ie, when ath9k does a full reset, actually _do_ a full reset by
> resetting the mac/wmac via the SoC lines.

So far, I could resolve the situation by doing ifconfig wlan0 down ; ifconfig
wlan0 up. But I don't know if this problem will be triggered much faster
in "more hostile" environments (wherever this comes from).

>
> It could also be the RTC state. Maybe it's found its way into network
> sleep (ie, the force wake bits aren't set right.) Or something PLL
> related isn't programmed correctly. Dumping the RTC registers would be
> helpful.

I'm not sure which registers you mean exactly, but I have looked through
the regdump, compared to ath9k's reg.h and annotated it a bit (see below).
As I lack the Chip Specification, maybe some QCA/Atheros dev can help me
on that?

BTW, I've uploaded the full dumps in case anyone is interested [1]. node200
is the broken one, node201 is working at that moment (for comparison).

Thanks,
Simon

[1] http://packetmixer.de/ar9330_deadbeef/

#### Some PLL stuff:

#define AR_CH0_BB_DPLL1 0x16180
0x016180 0x08400000
#define AR_CH0_BB_DPLL2 0x16184
0x016184 0x18e90f00
#define AR_CH0_BB_DPLL3 0x16188
0x016188 0x00800000
#define AR_CH0_DDR_DPLL2 0x16244
0x016244 0x18e82f01
#define AR_CH0_DDR_DPLL3 0x16248
0x016248 0x008024d0
0x01624c 0xfc4d78b0

#### RTC Registers, annotated:

0x007000 0x00000002
0x007004 0x00000000
0x007008 0x00000000
0x00700c 0x00000001
0x007010 0x00000022
0x007014 0x0000142c
0x007018 0x00000500
0x00701c 0x00000001
0x007020 0x00000000
0x007024 0x00000000
0x007028 0x00000000
0x00702c 0x00000000
0x007030 0x00000002
#define AR_RTC_KEEP_AWAKE 0x7034
0x007034 0x00000002
0x007038 0x000002f8
0x00703c 0x00e04a3d
#define AR_RTC_RESET ((AR_SREV_9100(ah)) ? (AR_RTC_BASE + 0x0040) : 0x7040
#define AR_RTC_RESET_EN (0x00000001)
0x007040 0x00000001
#define AR_RTC_STATUS ((AR_SREV_9100(ah)) ? (AR_RTC_BASE + 0x0044) : 0x7044)
#define AR_RTC_STATUS_M ((AR_SREV_9100(ah)) ? 0x0000003f : 0x0000000f)
#define AR_RTC_PM_STATUS_M 0x0000000f
#define AR_RTC_STATUS_SHUTDOWN 0x00000001
#define AR_RTC_STATUS_ON 0x00000002
#define AR_RTC_STATUS_SLEEP 0x00000004
#define AR_RTC_STATUS_WAKEUP 0x00000008
0x007044 0x00000002
--> so RTC Status is "on"?

#define AR_RTC_SLEEP_CLK ((AR_SREV_9100(ah)) ? (AR_RTC_BASE + 0x0048) : 0x7048)
#define AR_RTC_FORCE_DERIVED_CLK 0x2
#define AR_RTC_FORCE_SWREG_PRD 0x00000004

0x007048 0x00000002

#define AR_RTC_FORCE_WAKE ((AR_SREV_9100(ah)) ? (AR_RTC_BASE + 0x004c) : 0x704c)
#define AR_RTC_FORCE_WAKE_EN 0x00000001
#define AR_RTC_FORCE_WAKE_ON_INT 0x00000002
0x00704c 0x00000003

--> both on, looks good?

#define AR_RTC_INTR_CAUSE ((AR_SREV_9100(ah)) ? (AR_RTC_BASE + 0x0050) : 0x7050)
0x007050 0x00000000
#define AR_RTC_INTR_ENABLE ((AR_SREV_9100(ah)) ? (AR_RTC_BASE + 0x0054) : 0x7054)
0x007054 0x00000000
#define AR_RTC_INTR_MASK ((AR_SREV_9100(ah)) ? (AR_RTC_BASE + 0x0058) : 0x7058)
0x007058 0x00000000

Attachments:

(No filename) (3.46 kB)
signature.asc (198.00 B)
Digital signature Download all attachments

2012-08-23 15:28:01

by Felix Fietkau

[permalink] [raw]

Subject: Re: [OpenWrt-Devel] AR9330 hornet board stops beaconing after a few days (0xdeadbeef)

On 2012-09-13 6:51 PM, Simon Wunderlich wrote:
> ath9k fellows,
>
> as it seems no one could find the cause for this problem so far. I'd therefore
> like to create a workaround by checking one/some registers for 0xdeadbeef and
> reset the chip if this is found.
>
> Can anyone recommend a register which should never go 0xdeadbeef in a normal case?
>
> From what i've seen, AR_CFG (0x0014) might be a good choice. My regdumps say:
>
> bad regdump: 0x000014 0xdeadbeef
> good regdump: 0x000014 0x0008010a
>
> Unfortunately I don't have documentation to find out if this register can ever
> go deadbeef in a normal case. :)
>
> What do you think? (of course, a proper solution is still appreciated ...)
If ah->power_mode == ATH9K_PM_AWAKE, then no regularly used register may
return 0xdeadbeef. You can use a combination of the register and the
power mode as an indicator.

- Felix

2012-09-23 10:04:51

by Sven Eckelmann

[permalink] [raw]

Subject: Re: Re: [RFC] ath9k: Work around complete stuck of hw

On Friday 14 September 2012 13:33:31 Felix Fietkau wrote:
[..]
> > ---
> > This check is currently tested. This takes quite a long time and maybe
> > someone with more knowledge of atheros devices can check whether this one
> > is completely and utterly wrong.
> >
> > The type RESET_TYPE_FATAL_INT was chosen in this test to allow us to see
> > whether this condition was already true by reading from
> > /sys/kernel/debug/ieee80211/phy0/ath9k/reset
>
> Your debug patch should not be silent when it resets the hw. We need
> to make sure that this bug gets fixed properly. If my patch below does
> not fix it, then at least add a WARN_ON to ensure that we don't just
> hide the bug and move on.

It is not silent for me and is not a patch that should be applied to the
upstream kernel. Just wanted to ask whether this is the right place and
approach

> Somebody on the openwrt-devel list pointed out that there is some code
> missing in the ar933x wmac reset function. Please try this patch (apply
> it to your kernel tree):

Neither mine nor your patch has an effect. One node is now again dead after ~8
1/2 days. It looks like the check isn't done (correctly). I will investigate
this further tomorrow.

Kind regards,
Sven

Attachments:

signature.asc (836.00 B)
This is a digitally signed message part.

2012-09-02 20:00:53

On Wednesday 05 September 2012 07:08:47 Adrian Chadd wrote:
> I don't think it's a stuck beacon WAR issue here; 0xdeadbeef really
> does read like the MAC has gone to sleep.
>
> Where's the complete register dump hiding?

It was not sent to the public mailing lists due to the size. You will receive
the dump mentioned in the first mail in some minutes.

Kind regards,
Sven

Attachments:

signature.asc (836.00 B)
This is a digitally signed message part.

2013-11-20 20:50:01

by Bastian Bittorf

[permalink] [raw]

Subject: Re: [OpenWrt-Devel] AR9330 hornet board stops beaconing after a few days (0xdeadbeef)

* Simon Wunderlich <[email protected]> [19.09.2012 11:25]:
> just to bump again - isn't there anyone who can give us some hint or
> clue regarding these 0xdeadbeef entries in the registers? Couldn't find
> a connection to other reports deadbeef on the mailing list so far.

it doesnt make sense to debug such an old firmware (r31729).
first upgrade to trunk (~r38885), then report again. bye, bastian

2013-11-20 21:08:23

by Sven Eckelmann

[permalink] [raw]

Subject: Re: [OpenWrt-Devel] AR9330 hornet board stops beaconing after a few days (0xdeadbeef)

On Wednesday 20 November 2013 21:57:38 Bastian Bittorf wrote:
> * Simon Wunderlich <[email protected]> [19.09.2012
11:25]:
> > just to bump again - isn't there anyone who can give us some hint or
> > clue regarding these 0xdeadbeef entries in the registers? Couldn't find
> > a connection to other reports deadbeef on the mailing list so far.
>
> it doesnt make sense to debug such an old firmware (r31729).
> first upgrade to trunk (~r38885), then report again. bye, bastian

Doesn't make a lot of sense to reply to such an old thread.

Kind regards,
Sven