2012-07-18 12:03:30

by Bastian Bittorf

[permalink] [raw]
Subject: WRT54g / b43 / mac802.11 BREAKTHROUGH

hi devs!

yesterday we had a breaktrough debugging b43 in our hackspace maschinenraum/m18[1,2]
at weimar.freifunk.net[3,4] - since a long time our darling wrt54g suffers from a
hanging wifi and bad performance[5], but the workaround is easy: now it's up to
you to fix the rootcause.

our testsetup, where we could _reproduce_ reliably a stopping TX is like this:

laptop ---lan--- "A"-wrt54g/adhoc ~~~ air ~~~ "B"-anyrouter/adhoc

now make a testdownload with the laptop from B.
wireless will stop working after ~10 seconds.

wifi up will reanimate and our freifunk-cron.minutely-check
will do this automagically. (read further, this is not the solution)

we tried to limit the rateset to e.g. lower rates, but this does NOT change
the behaviour. what works is: define a rateset on BOTH router which makes
it impossible to change the band, e.g.:

iw dev $WIFIDEV set bitrates legacy-2.4 1 2 5.5 11
OR
iw dev $WIFIDEV set bitrates legacy-2.4 6 9 12 18 24 36 48 54

now we had a great performance, 10 Gigabytes of wireless transfer,
no stopping TX anymore and an empty box of beer. three things to do now:

1) why does a band change (can be seen through minstrel) is a problem?

2) we have to make in config-option to force a band, or a rateset:
e.g. uci set wireless.radio0.hwmode=11g
e.g. uci set wireless.radio0.bitrates='6 9 12 18 24 36 48 54'

3) spread to word:
there is a great frustration in the community about b43,
but the old drivers _have_ to die, it's about time - really.

thanks for your work,
bye storchi, andi, bastian + m18 crew

[1] http://blog.maschinenraum.tk/
[2] http://blog.maschinenraum.tk/2012/07/15/bitcoin-vending-machine-exchange-euro-coins-for-bitcoin-wallets/
[3] http://wireless.subsignal.org
[4] http://wireless.subsignal.org/index.php?title=Main_Page_en
[5] https://dev.openwrt.org/ticket/7552



2012-07-19 01:41:06

by Gábor Stefanik

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

On Wed, Jul 18, 2012 at 1:56 PM, Bastian Bittorf <[email protected]> wrote:
> hi devs!
>
> yesterday we had a breaktrough debugging b43 in our hackspace maschinenraum/m18[1,2]
> at weimar.freifunk.net[3,4] - since a long time our darling wrt54g suffers from a
> hanging wifi and bad performance[5], but the workaround is easy: now it's up to
> you to fix the rootcause.
>
> our testsetup, where we could _reproduce_ reliably a stopping TX is like this:
>
> laptop ---lan--- "A"-wrt54g/adhoc ~~~ air ~~~ "B"-anyrouter/adhoc
>
> now make a testdownload with the laptop from B.
> wireless will stop working after ~10 seconds.
>
> wifi up will reanimate and our freifunk-cron.minutely-check
> will do this automagically. (read further, this is not the solution)
>
> we tried to limit the rateset to e.g. lower rates, but this does NOT change
> the behaviour. what works is: define a rateset on BOTH router which makes
> it impossible to change the band, e.g.:
>
> iw dev $WIFIDEV set bitrates legacy-2.4 1 2 5.5 11
> OR
> iw dev $WIFIDEV set bitrates legacy-2.4 6 9 12 18 24 36 48 54
>
> now we had a great performance, 10 Gigabytes of wireless transfer,
> no stopping TX anymore and an empty box of beer. three things to do now:
>
> 1) why does a band change (can be seen through minstrel) is a problem?

Because IIRC b43 doesn't support anything other than 2.4GHz at all.

>
> 2) we have to make in config-option to force a band, or a rateset:
> e.g. uci set wireless.radio0.hwmode=11g
> e.g. uci set wireless.radio0.bitrates='6 9 12 18 24 36 48 54'
>
> 3) spread to word:
> there is a great frustration in the community about b43,
> but the old drivers _have_ to die, it's about time - really.
>
> thanks for your work,
> bye storchi, andi, bastian + m18 crew
>
> [1] http://blog.maschinenraum.tk/
> [2] http://blog.maschinenraum.tk/2012/07/15/bitcoin-vending-machine-exchange-euro-coins-for-bitcoin-wallets/
> [3] http://wireless.subsignal.org
> [4] http://wireless.subsignal.org/index.php?title=Main_Page_en
> [5] https://dev.openwrt.org/ticket/7552
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html



--
Vista: [V]iruses, [I]ntruders, [S]pyware, [T]rojans and [A]dware. :-)

2012-07-20 09:06:34

by Bastian Bittorf

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

> > 1) why does a band change (can be seen through minstrel) is a
> problem?
>
> Because IIRC b43 doesn't support anything other than 2.4GHz at
> all.

8-) sorry, i used the wrong term "band":
(fast) changing between b and g rates are the problem.

you can easily reproduce, when you force this by:

iw dev $WIFIDEV set bitrates legacy-2.4 11 12

("use only b-rate 11 and g-rate 12")
So minstrel will happily change/try both, wifi will stop.
The same with:

iw dev $WIFIDEV set bitrates legacy-2.4 5.5 6

bye, Bastian


2012-07-20 09:50:10

by Bastian Bittorf

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

> What chip is in your WRT54G?

We tested both:
4318 (Buffalo HP54G) and
4306 (Linksys WRT54GL)

bye, bastian


2012-07-19 07:25:41

by Andreas Bräu

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

>>
>> 1) why does a band change (can be seen through minstrel) is a problem?
>
> Because IIRC b43 doesn't support anything other than 2.4GHz at all.

It's not a change between 2.4 and 5 GHz.
Wifi stops working if the bitrate changes between 11b (1, 2, 5.5, 11) and 11g (6, 9, 12, 18, 24, 36, 48, 54) rates

So if we tell the system to only use 11g rates, wifi doesn't stop.

Andi

2012-07-19 14:59:47

by Larry Finger

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

On 07/19/2012 02:18 AM, Andreas Br?u wrote:
>>>
>>> 1) why does a band change (can be seen through minstrel) is a problem?
>>
>> Because IIRC b43 doesn't support anything other than 2.4GHz at all.
>
> It's not a change between 2.4 and 5 GHz.
> Wifi stops working if the bitrate changes between 11b (1, 2, 5.5, 11) and 11g (6, 9, 12, 18, 24, 36, 48, 54) rates
>
> So if we tell the system to only use 11g rates, wifi doesn't stop.

Andi,

Rates for 11g include 1, 2, 5.5, and 11, as well as 6, 9, ... What you really
want to say is that it fails when it switches between CCCK and OFDM rates.

What chip is in your WRT54G?

Larry



2012-07-19 10:03:29

by npl

[permalink] [raw]
Subject: Re: [maschinenraum] WRT54g / b43 / mac802.11 BREAKTHROUGH

> It's not a change between 2.4 and 5 GHz.
> Wifi stops working if the bitrate changes between 11b (1, 2, 5.5, 11) and 11g (6, 9, 12, 18, 24, 36, 48, 54) rates
>
> So if we tell the system to only use 11g rates, wifi doesn't stop.

Dann laufen aber 11b Karten nicht mehr.

Thomas

2012-08-24 17:13:14

by Bastian Bittorf

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

> Do you have any info whether the failure occurs with the change
> from OFDM to
> CCK, or vice versa? I am wondering if the radio needs a dummy

i cannot answer this question - interesting idea.
i like to test your patch, simply do a dummy tx in both cases.

bye, bastian


2012-08-22 18:17:36

by Hauke Mehrtens

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

On 08/22/2012 07:49 PM, Rafał Miłecki wrote:
> 2012/8/22 Hauke Mehrtens <[email protected]>:
>> On 07/18/2012 01:56 PM, Bastian Bittorf wrote:
>>> hi devs!
>>>
>>> yesterday we had a breaktrough debugging b43 in our hackspace maschinenraum/m18[1,2]
>>> at weimar.freifunk.net[3,4] - since a long time our darling wrt54g suffers from a
>>> hanging wifi and bad performance[5], but the workaround is easy: now it's up to
>>> you to fix the rootcause.
>>>
>>> our testsetup, where we could _reproduce_ reliably a stopping TX is like this:
>>>
>>> laptop ---lan--- "A"-wrt54g/adhoc ~~~ air ~~~ "B"-anyrouter/adhoc
>>>
>>> now make a testdownload with the laptop from B.
>>> wireless will stop working after ~10 seconds.
>>>
>>> wifi up will reanimate and our freifunk-cron.minutely-check
>>> will do this automagically. (read further, this is not the solution)
>>>
>>> we tried to limit the rateset to e.g. lower rates, but this does NOT change
>>> the behaviour. what works is: define a rateset on BOTH router which makes
>>> it impossible to change the band, e.g.:
>>>
>>> iw dev $WIFIDEV set bitrates legacy-2.4 1 2 5.5 11
>>> OR
>>> iw dev $WIFIDEV set bitrates legacy-2.4 6 9 12 18 24 36 48 54
>>>
>>> now we had a great performance, 10 Gigabytes of wireless transfer,
>>> no stopping TX anymore and an empty box of beer. three things to do now:
>>>
>>> 1) why does a band change (can be seen through minstrel) is a problem?
>>>
>>> 2) we have to make in config-option to force a band, or a rateset:
>>> e.g. uci set wireless.radio0.hwmode=11g
>>> e.g. uci set wireless.radio0.bitrates='6 9 12 18 24 36 48 54'
>>>
>>> 3) spread to word:
>>> there is a great frustration in the community about b43,
>>> but the old drivers _have_ to die, it's about time - really.
>>>
>>> thanks for your work,
>>> bye storchi, andi, bastian + m18 crew
>>>
>>> [1] http://blog.maschinenraum.tk/
>>> [2] http://blog.maschinenraum.tk/2012/07/15/bitcoin-vending-machine-exchange-euro-coins-for-bitcoin-wallets/
>>> [3] http://wireless.subsignal.org
>>> [4] http://wireless.subsignal.org/index.php?title=Main_Page_en
>>> [5] https://dev.openwrt.org/ticket/7552
>>
>> I tried this on some of my devices (BCM4318 G-PHY and BCM4322 N-PHY) and
>> it is easily reproducible on all tested devices when restricting the
>> rates to 11 and 12 MBit/s.
>>
>> I have the Broadcom device working as an Access point (on a MIPS SoC)
>> and a Laptop with an Intel Wifi device is connected to it. I generated
>> the traffic with iperf. If the Access Points sends the traffic the
>> problem occurs, if it is just receiving there is not problem.
>>
>> After the problem occurred b43_generate_txhdr is called rarely ( every
>> ~10 seconds) and I am not able to stay connected or connect to the
>> network any more. I am not familiar with the internal flow in b43 or
>> mac80211 could someone give me a hint where to look.
>>
>> I can not see any special changes between CCK and OFDM rates before it
>> goes down there are many changes without a problem before it goes down.
>>
>> Currently I do not have a Broadcom wifi card running in a x86 device
>> just mips devices could someone try to reproduce the problem on a x86
>> device.
>>
>> I added the b43 mailing list and Rafał.
>
> Does your kernel include
>
> commit bad6919469662b7c92bc6353642aaaa777b36bac
> Author: [email protected] <[email protected]>
> Date: Fri Dec 16 18:34:56 2011 +0100
>
> b43: avoid packet losses in the dma worker code.
>
> ? I have a lot of things on my TODO list, including that. Right now I
> just want to focus on BCM4706 support, which slowly moves into
> mainline.
>

This patch is included. I used compat-wireless-2012-07-16 plus some more
backport patches for b43.

2012-08-22 17:04:01

by G.W. Haywood

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

Hi there,

On Wed, 22 Aug 2012, Hauke Mehrtens wrote:

> ... If the Access Points sends the traffic the
> problem occurs, if it is just receiving there is not problem.

Interesting. I have not specifically tested my Linksys devices by
sending files from client to access point and vice-versa, but I can
say that the problem exists (well at least, a problem exists) when the
two stations are in ad-hoc mode, and so neither of them is acting as
an access point. The symptoms are the same: at high loads (megabits
per second), the wireless communication quickly fails (a few seconds)
and manual intervention is then required to recover the wireless link
by reconfiguring one or other of the interfaces. To recap, this is
OpenWRT built last week from trunk runing on Linksys WRT54GSv1.1.

The same equipment appears to function perfectly well at high loads
when using other firmware (Tomato version 1.28).

Unfortunately I do not have Broadcom devices which I can test on x86
but if someone can suggest suitable cards I will consider a purchase.
There are x86 boxes here which I can use for testing. It is certain
that I would need a few pointers to build a system which will perform
tests likely to provide useful information about the problem but I am
very familiar with the "./configure/make/make install" routine.

--

73,
Ged.

2012-08-22 17:49:36

by Rafał Miłecki

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

2012/8/22 Hauke Mehrtens <[email protected]>:
> On 07/18/2012 01:56 PM, Bastian Bittorf wrote:
>> hi devs!
>>
>> yesterday we had a breaktrough debugging b43 in our hackspace maschinenraum/m18[1,2]
>> at weimar.freifunk.net[3,4] - since a long time our darling wrt54g suffers from a
>> hanging wifi and bad performance[5], but the workaround is easy: now it's up to
>> you to fix the rootcause.
>>
>> our testsetup, where we could _reproduce_ reliably a stopping TX is like this:
>>
>> laptop ---lan--- "A"-wrt54g/adhoc ~~~ air ~~~ "B"-anyrouter/adhoc
>>
>> now make a testdownload with the laptop from B.
>> wireless will stop working after ~10 seconds.
>>
>> wifi up will reanimate and our freifunk-cron.minutely-check
>> will do this automagically. (read further, this is not the solution)
>>
>> we tried to limit the rateset to e.g. lower rates, but this does NOT change
>> the behaviour. what works is: define a rateset on BOTH router which makes
>> it impossible to change the band, e.g.:
>>
>> iw dev $WIFIDEV set bitrates legacy-2.4 1 2 5.5 11
>> OR
>> iw dev $WIFIDEV set bitrates legacy-2.4 6 9 12 18 24 36 48 54
>>
>> now we had a great performance, 10 Gigabytes of wireless transfer,
>> no stopping TX anymore and an empty box of beer. three things to do now:
>>
>> 1) why does a band change (can be seen through minstrel) is a problem?
>>
>> 2) we have to make in config-option to force a band, or a rateset:
>> e.g. uci set wireless.radio0.hwmode=11g
>> e.g. uci set wireless.radio0.bitrates='6 9 12 18 24 36 48 54'
>>
>> 3) spread to word:
>> there is a great frustration in the community about b43,
>> but the old drivers _have_ to die, it's about time - really.
>>
>> thanks for your work,
>> bye storchi, andi, bastian + m18 crew
>>
>> [1] http://blog.maschinenraum.tk/
>> [2] http://blog.maschinenraum.tk/2012/07/15/bitcoin-vending-machine-exchange-euro-coins-for-bitcoin-wallets/
>> [3] http://wireless.subsignal.org
>> [4] http://wireless.subsignal.org/index.php?title=Main_Page_en
>> [5] https://dev.openwrt.org/ticket/7552
>
> I tried this on some of my devices (BCM4318 G-PHY and BCM4322 N-PHY) and
> it is easily reproducible on all tested devices when restricting the
> rates to 11 and 12 MBit/s.
>
> I have the Broadcom device working as an Access point (on a MIPS SoC)
> and a Laptop with an Intel Wifi device is connected to it. I generated
> the traffic with iperf. If the Access Points sends the traffic the
> problem occurs, if it is just receiving there is not problem.
>
> After the problem occurred b43_generate_txhdr is called rarely ( every
> ~10 seconds) and I am not able to stay connected or connect to the
> network any more. I am not familiar with the internal flow in b43 or
> mac80211 could someone give me a hint where to look.
>
> I can not see any special changes between CCK and OFDM rates before it
> goes down there are many changes without a problem before it goes down.
>
> Currently I do not have a Broadcom wifi card running in a x86 device
> just mips devices could someone try to reproduce the problem on a x86
> device.
>
> I added the b43 mailing list and Rafał.

Does your kernel include

commit bad6919469662b7c92bc6353642aaaa777b36bac
Author: [email protected] <[email protected]>
Date: Fri Dec 16 18:34:56 2011 +0100

b43: avoid packet losses in the dma worker code.

? I have a lot of things on my TODO list, including that. Right now I
just want to focus on BCM4706 support, which slowly moves into
mainline.

--
Rafał

2012-08-22 16:21:45

by Hauke Mehrtens

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

On 07/18/2012 01:56 PM, Bastian Bittorf wrote:
> hi devs!
>
> yesterday we had a breaktrough debugging b43 in our hackspace maschinenraum/m18[1,2]
> at weimar.freifunk.net[3,4] - since a long time our darling wrt54g suffers from a
> hanging wifi and bad performance[5], but the workaround is easy: now it's up to
> you to fix the rootcause.
>
> our testsetup, where we could _reproduce_ reliably a stopping TX is like this:
>
> laptop ---lan--- "A"-wrt54g/adhoc ~~~ air ~~~ "B"-anyrouter/adhoc
>
> now make a testdownload with the laptop from B.
> wireless will stop working after ~10 seconds.
>
> wifi up will reanimate and our freifunk-cron.minutely-check
> will do this automagically. (read further, this is not the solution)
>
> we tried to limit the rateset to e.g. lower rates, but this does NOT change
> the behaviour. what works is: define a rateset on BOTH router which makes
> it impossible to change the band, e.g.:
>
> iw dev $WIFIDEV set bitrates legacy-2.4 1 2 5.5 11
> OR
> iw dev $WIFIDEV set bitrates legacy-2.4 6 9 12 18 24 36 48 54
>
> now we had a great performance, 10 Gigabytes of wireless transfer,
> no stopping TX anymore and an empty box of beer. three things to do now:
>
> 1) why does a band change (can be seen through minstrel) is a problem?
>
> 2) we have to make in config-option to force a band, or a rateset:
> e.g. uci set wireless.radio0.hwmode=11g
> e.g. uci set wireless.radio0.bitrates='6 9 12 18 24 36 48 54'
>
> 3) spread to word:
> there is a great frustration in the community about b43,
> but the old drivers _have_ to die, it's about time - really.
>
> thanks for your work,
> bye storchi, andi, bastian + m18 crew
>
> [1] http://blog.maschinenraum.tk/
> [2] http://blog.maschinenraum.tk/2012/07/15/bitcoin-vending-machine-exchange-euro-coins-for-bitcoin-wallets/
> [3] http://wireless.subsignal.org
> [4] http://wireless.subsignal.org/index.php?title=Main_Page_en
> [5] https://dev.openwrt.org/ticket/7552

I tried this on some of my devices (BCM4318 G-PHY and BCM4322 N-PHY) and
it is easily reproducible on all tested devices when restricting the
rates to 11 and 12 MBit/s.

I have the Broadcom device working as an Access point (on a MIPS SoC)
and a Laptop with an Intel Wifi device is connected to it. I generated
the traffic with iperf. If the Access Points sends the traffic the
problem occurs, if it is just receiving there is not problem.

After the problem occurred b43_generate_txhdr is called rarely ( every
~10 seconds) and I am not able to stay connected or connect to the
network any more. I am not familiar with the internal flow in b43 or
mac80211 could someone give me a hint where to look.

I can not see any special changes between CCK and OFDM rates before it
goes down there are many changes without a problem before it goes down.

Currently I do not have a Broadcom wifi card running in a x86 device
just mips devices could someone try to reproduce the problem on a x86
device.

I added the b43 mailing list and Rafał.

Hauke

2012-08-22 18:45:22

by Larry Finger

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

On 08/22/2012 11:21 AM, Hauke Mehrtens wrote:
> On 07/18/2012 01:56 PM, Bastian Bittorf wrote:
>> hi devs!
>>
>> yesterday we had a breaktrough debugging b43 in our hackspace maschinenraum/m18[1,2]
>> at weimar.freifunk.net[3,4] - since a long time our darling wrt54g suffers from a
>> hanging wifi and bad performance[5], but the workaround is easy: now it's up to
>> you to fix the rootcause.
>>
>> our testsetup, where we could _reproduce_ reliably a stopping TX is like this:
>>
>> laptop ---lan--- "A"-wrt54g/adhoc ~~~ air ~~~ "B"-anyrouter/adhoc
>>
>> now make a testdownload with the laptop from B.
>> wireless will stop working after ~10 seconds.
>>
>> wifi up will reanimate and our freifunk-cron.minutely-check
>> will do this automagically. (read further, this is not the solution)
>>
>> we tried to limit the rateset to e.g. lower rates, but this does NOT change
>> the behaviour. what works is: define a rateset on BOTH router which makes
>> it impossible to change the band, e.g.:
>>
>> iw dev $WIFIDEV set bitrates legacy-2.4 1 2 5.5 11
>> OR
>> iw dev $WIFIDEV set bitrates legacy-2.4 6 9 12 18 24 36 48 54
>>
>> now we had a great performance, 10 Gigabytes of wireless transfer,
>> no stopping TX anymore and an empty box of beer. three things to do now:
>>
>> 1) why does a band change (can be seen through minstrel) is a problem?
>>
>> 2) we have to make in config-option to force a band, or a rateset:
>> e.g. uci set wireless.radio0.hwmode=11g
>> e.g. uci set wireless.radio0.bitrates='6 9 12 18 24 36 48 54'
>>
>> 3) spread to word:
>> there is a great frustration in the community about b43,
>> but the old drivers _have_ to die, it's about time - really.
>>
>> thanks for your work,
>> bye storchi, andi, bastian + m18 crew
>>
>> [1] http://blog.maschinenraum.tk/
>> [2] http://blog.maschinenraum.tk/2012/07/15/bitcoin-vending-machine-exchange-euro-coins-for-bitcoin-wallets/
>> [3] http://wireless.subsignal.org
>> [4] http://wireless.subsignal.org/index.php?title=Main_Page_en
>> [5] https://dev.openwrt.org/ticket/7552
>
> I tried this on some of my devices (BCM4318 G-PHY and BCM4322 N-PHY) and
> it is easily reproducible on all tested devices when restricting the
> rates to 11 and 12 MBit/s.
>
> I have the Broadcom device working as an Access point (on a MIPS SoC)
> and a Laptop with an Intel Wifi device is connected to it. I generated
> the traffic with iperf. If the Access Points sends the traffic the
> problem occurs, if it is just receiving there is not problem.
>
> After the problem occurred b43_generate_txhdr is called rarely ( every
> ~10 seconds) and I am not able to stay connected or connect to the
> network any more. I am not familiar with the internal flow in b43 or
> mac80211 could someone give me a hint where to look.
>
> I can not see any special changes between CCK and OFDM rates before it
> goes down there are many changes without a problem before it goes down.
>
> Currently I do not have a Broadcom wifi card running in a x86 device
> just mips devices could someone try to reproduce the problem on a x86
> device.
>
> I added the b43 mailing list and Rafał.

Hauke or Bastian,

Do you have any info whether the failure occurs with the change from OFDM to
CCK, or vice versa? I am wondering if the radio needs a dummy transmission when
the switch happens. I will try to put together a patch to test that hypothesis.

Larry



2012-08-24 17:18:44

by Bastian Bittorf

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

> How do I monitor this particular channel with an other wifi card to
> get
> most of the packages, with aircrack I am unable to set the
> channel?

the simple approach is installing horst on another router
or your pc. ofcourse you get a full dump with wireshark
sniffing on your monitor-interface.

bye, bastian

2012-08-22 21:19:19

by Larry Finger

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

On 08/22/2012 04:05 PM, Hauke Mehrtens wrote:
> On 08/22/2012 08:45 PM, Larry Finger wrote:
>> On 08/22/2012 11:21 AM, Hauke Mehrtens wrote:
>>> On 07/18/2012 01:56 PM, Bastian Bittorf wrote:
>>>> hi devs!
>>>>
>>>> yesterday we had a breaktrough debugging b43 in our hackspace
>>>> maschinenraum/m18[1,2]
>>>> at weimar.freifunk.net[3,4] - since a long time our darling wrt54g
>>>> suffers from a
>>>> hanging wifi and bad performance[5], but the workaround is easy: now
>>>> it's up to
>>>> you to fix the rootcause.
>>>>
>>>> our testsetup, where we could _reproduce_ reliably a stopping TX is
>>>> like this:
>>>>
>>>> laptop ---lan--- "A"-wrt54g/adhoc ~~~ air ~~~ "B"-anyrouter/adhoc
>>>>
>>>> now make a testdownload with the laptop from B.
>>>> wireless will stop working after ~10 seconds.
>>>>
>>>> wifi up will reanimate and our freifunk-cron.minutely-check
>>>> will do this automagically. (read further, this is not the solution)
>>>>
>>>> we tried to limit the rateset to e.g. lower rates, but this does NOT
>>>> change
>>>> the behaviour. what works is: define a rateset on BOTH router which
>>>> makes
>>>> it impossible to change the band, e.g.:
>>>>
>>>> iw dev $WIFIDEV set bitrates legacy-2.4 1 2 5.5 11
>>>> OR
>>>> iw dev $WIFIDEV set bitrates legacy-2.4 6 9 12 18 24 36 48 54
>>>>
>>>> now we had a great performance, 10 Gigabytes of wireless transfer,
>>>> no stopping TX anymore and an empty box of beer. three things to do now:
>>>>
>>>> 1) why does a band change (can be seen through minstrel) is a problem?
>>>>
>>>> 2) we have to make in config-option to force a band, or a rateset:
>>>> e.g. uci set wireless.radio0.hwmode=11g
>>>> e.g. uci set wireless.radio0.bitrates='6 9 12 18 24 36 48 54'
>>>>
>>>> 3) spread to word:
>>>> there is a great frustration in the community about b43,
>>>> but the old drivers _have_ to die, it's about time - really.
>>>>
>>>> thanks for your work,
>>>> bye storchi, andi, bastian + m18 crew
>>>>
>>>> [1] http://blog.maschinenraum.tk/
>>>> [2]
>>>> http://blog.maschinenraum.tk/2012/07/15/bitcoin-vending-machine-exchange-euro-coins-for-bitcoin-wallets/
>>>>
>>>> [3] http://wireless.subsignal.org
>>>> [4] http://wireless.subsignal.org/index.php?title=Main_Page_en
>>>> [5] https://dev.openwrt.org/ticket/7552
>>>
>>> I tried this on some of my devices (BCM4318 G-PHY and BCM4322 N-PHY) and
>>> it is easily reproducible on all tested devices when restricting the
>>> rates to 11 and 12 MBit/s.
>>>
>>> I have the Broadcom device working as an Access point (on a MIPS SoC)
>>> and a Laptop with an Intel Wifi device is connected to it. I generated
>>> the traffic with iperf. If the Access Points sends the traffic the
>>> problem occurs, if it is just receiving there is not problem.
>>>
>>> After the problem occurred b43_generate_txhdr is called rarely ( every
>>> ~10 seconds) and I am not able to stay connected or connect to the
>>> network any more. I am not familiar with the internal flow in b43 or
>>> mac80211 could someone give me a hint where to look.
>>>
>>> I can not see any special changes between CCK and OFDM rates before it
>>> goes down there are many changes without a problem before it goes down.
>>>
>>> Currently I do not have a Broadcom wifi card running in a x86 device
>>> just mips devices could someone try to reproduce the problem on a x86
>>> device.
>>>
>>> I added the b43 mailing list and Rafał.
>>
>> Hauke or Bastian,
>>
>> Do you have any info whether the failure occurs with the change from
>> OFDM to CCK, or vice versa? I am wondering if the radio needs a dummy
>> transmission when the switch happens. I will try to put together a patch
>> to test that hypothesis.
>>
>> Larry
>
> It is strange. The stop does not occur directly after changing from OFDM
> to CCK or vice versa. TX seams to still work, but the device does not
> receive anything any more.
>
> Here are some logs from my device:
> http://hauke-m.de/files/b43/wifi-stop/2012-08-22/b43-ap-block.txt
> Patch used:
> http://hauke-m.de/files/b43/wifi-stop/2012-08-22/b43-ap-block.patch
>
> "iw dev wlan0 set bitrates legacy-2.4 11 12" was issued at ~120.000000.
>
> How do I monitor this particular channel with an other wifi card to get
> most of the packages, with aircrack I am unable to set the channel?

Thanks for the log and the patch you used.

You should be able to use Kismet and set the channel; however, I usually let it
capture everything, and then use wireshark to analyze the pcap file. That way,
it is fairly easy to set filters to exclude the unwanted traffic from other AP
and STA sources.

I don't use wireshark to capture the data because the box I use does not have a
GUI desktop, and the command-line kismet is perfect for that setup.

Larry



2012-08-22 21:06:09

by Hauke Mehrtens

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

On 08/22/2012 08:45 PM, Larry Finger wrote:
> On 08/22/2012 11:21 AM, Hauke Mehrtens wrote:
>> On 07/18/2012 01:56 PM, Bastian Bittorf wrote:
>>> hi devs!
>>>
>>> yesterday we had a breaktrough debugging b43 in our hackspace
>>> maschinenraum/m18[1,2]
>>> at weimar.freifunk.net[3,4] - since a long time our darling wrt54g
>>> suffers from a
>>> hanging wifi and bad performance[5], but the workaround is easy: now
>>> it's up to
>>> you to fix the rootcause.
>>>
>>> our testsetup, where we could _reproduce_ reliably a stopping TX is
>>> like this:
>>>
>>> laptop ---lan--- "A"-wrt54g/adhoc ~~~ air ~~~ "B"-anyrouter/adhoc
>>>
>>> now make a testdownload with the laptop from B.
>>> wireless will stop working after ~10 seconds.
>>>
>>> wifi up will reanimate and our freifunk-cron.minutely-check
>>> will do this automagically. (read further, this is not the solution)
>>>
>>> we tried to limit the rateset to e.g. lower rates, but this does NOT
>>> change
>>> the behaviour. what works is: define a rateset on BOTH router which
>>> makes
>>> it impossible to change the band, e.g.:
>>>
>>> iw dev $WIFIDEV set bitrates legacy-2.4 1 2 5.5 11
>>> OR
>>> iw dev $WIFIDEV set bitrates legacy-2.4 6 9 12 18 24 36 48 54
>>>
>>> now we had a great performance, 10 Gigabytes of wireless transfer,
>>> no stopping TX anymore and an empty box of beer. three things to do now:
>>>
>>> 1) why does a band change (can be seen through minstrel) is a problem?
>>>
>>> 2) we have to make in config-option to force a band, or a rateset:
>>> e.g. uci set wireless.radio0.hwmode=11g
>>> e.g. uci set wireless.radio0.bitrates='6 9 12 18 24 36 48 54'
>>>
>>> 3) spread to word:
>>> there is a great frustration in the community about b43,
>>> but the old drivers _have_ to die, it's about time - really.
>>>
>>> thanks for your work,
>>> bye storchi, andi, bastian + m18 crew
>>>
>>> [1] http://blog.maschinenraum.tk/
>>> [2]
>>> http://blog.maschinenraum.tk/2012/07/15/bitcoin-vending-machine-exchange-euro-coins-for-bitcoin-wallets/
>>>
>>> [3] http://wireless.subsignal.org
>>> [4] http://wireless.subsignal.org/index.php?title=Main_Page_en
>>> [5] https://dev.openwrt.org/ticket/7552
>>
>> I tried this on some of my devices (BCM4318 G-PHY and BCM4322 N-PHY) and
>> it is easily reproducible on all tested devices when restricting the
>> rates to 11 and 12 MBit/s.
>>
>> I have the Broadcom device working as an Access point (on a MIPS SoC)
>> and a Laptop with an Intel Wifi device is connected to it. I generated
>> the traffic with iperf. If the Access Points sends the traffic the
>> problem occurs, if it is just receiving there is not problem.
>>
>> After the problem occurred b43_generate_txhdr is called rarely ( every
>> ~10 seconds) and I am not able to stay connected or connect to the
>> network any more. I am not familiar with the internal flow in b43 or
>> mac80211 could someone give me a hint where to look.
>>
>> I can not see any special changes between CCK and OFDM rates before it
>> goes down there are many changes without a problem before it goes down.
>>
>> Currently I do not have a Broadcom wifi card running in a x86 device
>> just mips devices could someone try to reproduce the problem on a x86
>> device.
>>
>> I added the b43 mailing list and Rafał.
>
> Hauke or Bastian,
>
> Do you have any info whether the failure occurs with the change from
> OFDM to CCK, or vice versa? I am wondering if the radio needs a dummy
> transmission when the switch happens. I will try to put together a patch
> to test that hypothesis.
>
> Larry

It is strange. The stop does not occur directly after changing from OFDM
to CCK or vice versa. TX seams to still work, but the device does not
receive anything any more.

Here are some logs from my device:
http://hauke-m.de/files/b43/wifi-stop/2012-08-22/b43-ap-block.txt
Patch used:
http://hauke-m.de/files/b43/wifi-stop/2012-08-22/b43-ap-block.patch

"iw dev wlan0 set bitrates legacy-2.4 11 12" was issued at ~120.000000.

How do I monitor this particular channel with an other wifi card to get
most of the packages, with aircrack I am unable to set the channel?

Hauke

2012-08-25 14:51:14

by Hauke Mehrtens

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

On 08/22/2012 11:19 PM, Larry Finger wrote:
> On 08/22/2012 04:05 PM, Hauke Mehrtens wrote:
>> On 08/22/2012 08:45 PM, Larry Finger wrote:
>>> On 08/22/2012 11:21 AM, Hauke Mehrtens wrote:
>>>> On 07/18/2012 01:56 PM, Bastian Bittorf wrote:
>>>>> hi devs!
>>>>>
>>>>> yesterday we had a breaktrough debugging b43 in our hackspace
>>>>> maschinenraum/m18[1,2]
>>>>> at weimar.freifunk.net[3,4] - since a long time our darling wrt54g
>>>>> suffers from a
>>>>> hanging wifi and bad performance[5], but the workaround is easy: now
>>>>> it's up to
>>>>> you to fix the rootcause.
>>>>>
>>>>> our testsetup, where we could _reproduce_ reliably a stopping TX is
>>>>> like this:
>>>>>
>>>>> laptop ---lan--- "A"-wrt54g/adhoc ~~~ air ~~~ "B"-anyrouter/adhoc
>>>>>
>>>>> now make a testdownload with the laptop from B.
>>>>> wireless will stop working after ~10 seconds.
>>>>>
>>>>> wifi up will reanimate and our freifunk-cron.minutely-check
>>>>> will do this automagically. (read further, this is not the solution)
>>>>>
>>>>> we tried to limit the rateset to e.g. lower rates, but this does NOT
>>>>> change
>>>>> the behaviour. what works is: define a rateset on BOTH router which
>>>>> makes
>>>>> it impossible to change the band, e.g.:
>>>>>
>>>>> iw dev $WIFIDEV set bitrates legacy-2.4 1 2 5.5 11
>>>>> OR
>>>>> iw dev $WIFIDEV set bitrates legacy-2.4 6 9 12 18 24 36 48 54
>>>>>
>>>>> now we had a great performance, 10 Gigabytes of wireless transfer,
>>>>> no stopping TX anymore and an empty box of beer. three things to do
>>>>> now:
>>>>>
>>>>> 1) why does a band change (can be seen through minstrel) is a problem?
>>>>>
>>>>> 2) we have to make in config-option to force a band, or a rateset:
>>>>> e.g. uci set wireless.radio0.hwmode=11g
>>>>> e.g. uci set wireless.radio0.bitrates='6 9 12 18 24 36 48 54'
>>>>>
>>>>> 3) spread to word:
>>>>> there is a great frustration in the community about b43,
>>>>> but the old drivers _have_ to die, it's about time - really.
>>>>>
>>>>> thanks for your work,
>>>>> bye storchi, andi, bastian + m18 crew
>>>>>
>>>>> [1] http://blog.maschinenraum.tk/
>>>>> [2]
>>>>> http://blog.maschinenraum.tk/2012/07/15/bitcoin-vending-machine-exchange-euro-coins-for-bitcoin-wallets/
>>>>>
>>>>>
>>>>> [3] http://wireless.subsignal.org
>>>>> [4] http://wireless.subsignal.org/index.php?title=Main_Page_en
>>>>> [5] https://dev.openwrt.org/ticket/7552
>>>>
>>>> I tried this on some of my devices (BCM4318 G-PHY and BCM4322 N-PHY)
>>>> and
>>>> it is easily reproducible on all tested devices when restricting the
>>>> rates to 11 and 12 MBit/s.
>>>>
>>>> I have the Broadcom device working as an Access point (on a MIPS SoC)
>>>> and a Laptop with an Intel Wifi device is connected to it. I generated
>>>> the traffic with iperf. If the Access Points sends the traffic the
>>>> problem occurs, if it is just receiving there is not problem.
>>>>
>>>> After the problem occurred b43_generate_txhdr is called rarely ( every
>>>> ~10 seconds) and I am not able to stay connected or connect to the
>>>> network any more. I am not familiar with the internal flow in b43 or
>>>> mac80211 could someone give me a hint where to look.
>>>>
>>>> I can not see any special changes between CCK and OFDM rates before it
>>>> goes down there are many changes without a problem before it goes down.
>>>>
>>>> Currently I do not have a Broadcom wifi card running in a x86 device
>>>> just mips devices could someone try to reproduce the problem on a x86
>>>> device.
>>>>
>>>> I added the b43 mailing list and Rafał.
>>>
>>> Hauke or Bastian,
>>>
>>> Do you have any info whether the failure occurs with the change from
>>> OFDM to CCK, or vice versa? I am wondering if the radio needs a dummy
>>> transmission when the switch happens. I will try to put together a patch
>>> to test that hypothesis.
>>>
>>> Larry
>>
>> It is strange. The stop does not occur directly after changing from OFDM
>> to CCK or vice versa. TX seams to still work, but the device does not
>> receive anything any more.
>>
>> Here are some logs from my device:
>> http://hauke-m.de/files/b43/wifi-stop/2012-08-22/b43-ap-block.txt
>> Patch used:
>> http://hauke-m.de/files/b43/wifi-stop/2012-08-22/b43-ap-block.patch
>>
>> "iw dev wlan0 set bitrates legacy-2.4 11 12" was issued at ~120.000000.
>>
>> How do I monitor this particular channel with an other wifi card to get
>> most of the packages, with aircrack I am unable to set the channel?
>
> Thanks for the log and the patch you used.
>
> You should be able to use Kismet and set the channel; however, I usually
> let it capture everything, and then use wireshark to analyze the pcap
> file. That way, it is fairly easy to set filters to exclude the unwanted
> traffic from other AP and STA sources.
>
> I don't use wireshark to capture the data because the box I use does not
> have a GUI desktop, and the command-line kismet is perfect for that setup.
>
> Larry

Hi Larry,

I did some further debugging and saw that the driver gets an interrupt
indicating some new received frames, but b43_dma_rx does not fetch any
new frames.

Here are some logs from my device:
http://hauke-m.de/files/b43/wifi-stop/2012-08-25/b43-ap-block-2.log
Patch used:
http://hauke-m.de/files/b43/wifi-stop/2012-08-25/b43-ap-block-2.patch

Sometime after 210 my wifi client disconnects from the network.

With PIO the AP goes down when trying to connect to it with a wifi client.

Hauke

2013-02-14 20:41:46

by Bastian Bittorf

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

* Bastian Bittorf <[email protected]> [18.07.2012 17:43]:
> hi devs!
>
> yesterday we had a breaktrough debugging b43 in our hackspace maschinenraum/m18[1,2]
> at weimar.freifunk.net[3,4] - since a long time our darling wrt54g suffers from a
> hanging wifi and bad performance[5], but the workaround is easy: now it's up to
> you to fix the rootcause.

[...]

another issues was found, see https://dev.openwrt.org/ticket/7552#comment:69

"...under high load and have found that probably some of the silent freezes
are due to overflow of the RX DMA buffer, seems like b43 does not
handlre such a situation at all."

https://dev.openwrt.org/attachment/ticket/7552/840-b43-workaround-rx-fifo-overflow.patch

the patch is working so far, but is only a workaround.
has somebody a better code-idea?

bye, bastian

2013-02-15 19:47:50

by Larry Finger

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

On 02/14/2013 02:35 PM, Bastian Bittorf wrote:
Bastian,

> * Bastian Bittorf <[email protected]> [18.07.2012 17:43]:
>> hi devs!
>>
>> yesterday we had a breaktrough debugging b43 in our hackspace maschinenraum/m18[1,2]
>> at weimar.freifunk.net[3,4] - since a long time our darling wrt54g suffers from a
>> hanging wifi and bad performance[5], but the workaround is easy: now it's up to
>> you to fix the rootcause.
>
> [...]
>
> another issues was found, see https://dev.openwrt.org/ticket/7552#comment:69
>
> "...under high load and have found that probably some of the silent freezes
> are due to overflow of the RX DMA buffer, seems like b43 does not
> handlre such a situation at all."
>
> https://dev.openwrt.org/attachment/ticket/7552/840-b43-workaround-rx-fifo-overflow.patch
>
> the patch is working so far, but is only a workaround.
> has somebody a better code-idea?

I am looking into this problem; however, as it has been several years since I
looked at the dma code, it might take a while. In the meantime, I have some
questions, and one thing for you to try.

How frequently do you hit the buffer overflow?

Is the problem caused by a load with high bursts, or is the high load relatively
constant?

Please change B43_RXRING_SLOTS in drivers/net/wireless/b43.h from 64 to 128
while keeping your recovery patch in place. Unless you are extremely limited in
the memory, that should help. I think that change has helped on one of my 32-bit
systems. I still want to keep track of the minimum in the free slots and expose
that quantity in /sys, but first we should see if changing the number of slots
helps you.

Larry


2013-02-18 02:16:43

by Larry Finger

[permalink] [raw]
Subject: Re: WRT54g / b43 / mac802.11 BREAKTHROUGH

On 02/14/2013 02:35 PM, Bastian Bittorf wrote:
Bastian,

> * Bastian Bittorf <[email protected]> [18.07.2012 17:43]:
>> hi devs!
>>
>> yesterday we had a breaktrough debugging b43 in our hackspace maschinenraum/m18[1,2]
>> at weimar.freifunk.net[3,4] - since a long time our darling wrt54g suffers from a
>> hanging wifi and bad performance[5], but the workaround is easy: now it's up to
>> you to fix the rootcause.
>
> [...]
>
> another issues was found, see https://dev.openwrt.org/ticket/7552#comment:69
>
> "...under high load and have found that probably some of the silent freezes
> are due to overflow of the RX DMA buffer, seems like b43 does not
> handlre such a situation at all."
>
> https://dev.openwrt.org/attachment/ticket/7552/840-b43-workaround-rx-fifo-overflow.patch
>
> the patch is working so far, but is only a workaround.
> has somebody a better code-idea?

Changing the maximum number of slots is certainly the correct thing to do.
Making that change on a netbook that would hang on high throughput cured its
problem. After some heavy usage, I found the following in the dmesg output:

b43-phy0 debug: DMA-64 rx_ring: Used slots 109/128, Failed frames 0/0 = 0.0%,
===================

Clearly 64 slots is not sufficient. I am currently testing to see how large it
can be made as I am not sure that 128 gives enough free space.

Larry