2020-01-29 18:58:47

by Marlon Smith

[permalink] [raw]
Subject: Strange performance issue when using two devices at once

Hi everyone,

I have two RT5370 devices connected to the same access point. Both
devices are very slow, but the instant I disconnect one device, the
other speeds up by a factor of 10.

The really strange part is that one device will perform slowly even if
the other device is basically idle! I've confirmed this with a packet
sniffer.

I've been trying to do some debugging, and I've found that when both
devices are connected to the access point, they report a large number
of duplicate frames. I added some debug output
in ieee80211_rx_h_check_dup() to confirm that this only happens while
both devices are connected. The packet sniffer also shows a large
number of retries while this is occurring.

Using backports 5.3-rc4 for this, but also tested on 4.14-rc2.

I did post about this previously on this mailing list (RT5370
performance issues), but I thought I'd post again with this new
information and more descriptive title. I'm a little bit stuck on this
for a while now, so any ideas are much appreciated.

Thanks!

Marlon


2020-01-29 18:58:47

by Ben Greear

[permalink] [raw]
Subject: Re: Strange performance issue when using two devices at once

On 1/29/20 10:39 AM, Marlon Smith wrote:
> Hi everyone,
>
> I have two RT5370 devices connected to the same access point. Both
> devices are very slow, but the instant I disconnect one device, the
> other speeds up by a factor of 10.

Out of curiosity, are both of the RT5370 used on the same client device?

Did you check that they have unique MAC addresses?

Thanks,
Ben

>
> The really strange part is that one device will perform slowly even if
> the other device is basically idle! I've confirmed this with a packet
> sniffer.
>
> I've been trying to do some debugging, and I've found that when both
> devices are connected to the access point, they report a large number
> of duplicate frames. I added some debug output
> in ieee80211_rx_h_check_dup() to confirm that this only happens while
> both devices are connected. The packet sniffer also shows a large
> number of retries while this is occurring.
>
> Using backports 5.3-rc4 for this, but also tested on 4.14-rc2.
>
> I did post about this previously on this mailing list (RT5370
> performance issues), but I thought I'd post again with this new
> information and more descriptive title. I'm a little bit stuck on this
> for a while now, so any ideas are much appreciated.
>
> Thanks!
>
> Marlon
>


--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com

2020-01-29 19:32:55

by Marlon Smith

[permalink] [raw]
Subject: Re: Strange performance issue when using two devices at once

On Wed, 2020-01-29 at 10:42 -0800, Ben Greear wrote:
> On 1/29/20 10:39 AM, Marlon Smith wrote:
> >
> > Hi everyone,
> >
> > I have two RT5370 devices connected to the same access point. Both
> > devices are very slow, but the instant I disconnect one device, the
> > other speeds up by a factor of 10.
> Out of curiosity, are both of the RT5370 used on the same client
> device?
> > Did you check that they have unique MAC addresses?
> > Thanks,
> Ben
> > >
> >
> > The really strange part is that one device will perform slowly even
> > if
> > the other device is basically idle! I've confirmed this with a
> > packet
> > sniffer.
> >
> > I've been trying to do some debugging, and I've found that when
> > both
> > devices are connected to the access point, they report a large
> > number
> > of duplicate frames. I added some debug output
> > in ieee80211_rx_h_check_dup() to confirm that this only happens
> > while
> > both devices are connected. The packet sniffer also shows a large
> > number of retries while this is occurring.
> >
> > Using backports 5.3-rc4 for this, but also tested on 4.14-rc2.
> >
> > I did post about this previously on this mailing list (RT5370
> > performance issues), but I thought I'd post again with this new
> > information and more descriptive title. I'm a little bit stuck on
> > this
> > for a while now, so any ideas are much appreciated.
> >
> > Thanks!
> >
> > Marlon
> >

They are on separate devices, although the mac addresses are close.
70:F1:1C:2E:AF:B4 and
70:F1:1C:2E:AF:B6.

However, I have a third device 70:F1:1C:2E:AF:BB which performs well
and does not affect the performance of the other two.

2020-01-29 19:33:14

by Ben Greear

[permalink] [raw]
Subject: Re: Strange performance issue when using two devices at once

On 1/29/20 11:22 AM, Marlon Smith wrote:
> On Wed, 2020-01-29 at 10:42 -0800, Ben Greear wrote:
>> On 1/29/20 10:39 AM, Marlon Smith wrote:
>>>
>>> Hi everyone,
>>>
>>> I have two RT5370 devices connected to the same access point. Both
>>> devices are very slow, but the instant I disconnect one device, the
>>> other speeds up by a factor of 10.
>> Out of curiosity, are both of the RT5370 used on the same client
>> device?
>>> Did you check that they have unique MAC addresses?
>>> Thanks,
>> Ben
>>>>
>>>
>>> The really strange part is that one device will perform slowly even
>>> if
>>> the other device is basically idle! I've confirmed this with a
>>> packet
>>> sniffer.
>>>
>>> I've been trying to do some debugging, and I've found that when
>>> both
>>> devices are connected to the access point, they report a large
>>> number
>>> of duplicate frames. I added some debug output
>>> in ieee80211_rx_h_check_dup() to confirm that this only happens
>>> while
>>> both devices are connected. The packet sniffer also shows a large
>>> number of retries while this is occurring.
>>>
>>> Using backports 5.3-rc4 for this, but also tested on 4.14-rc2.
>>>
>>> I did post about this previously on this mailing list (RT5370
>>> performance issues), but I thought I'd post again with this new
>>> information and more descriptive title. I'm a little bit stuck on
>>> this
>>> for a while now, so any ideas are much appreciated.
>>>
>>> Thanks!
>>>
>>> Marlon
>>>
>
> They are on separate devices, although the mac addresses are close.
> 70:F1:1C:2E:AF:B4 and
> 70:F1:1C:2E:AF:B6.
>
> However, I have a third device 70:F1:1C:2E:AF:BB which performs well
> and does not affect the performance of the other two.
>

Have you tried a different AP?

And also tried using the exact same MAC addresses configured on a different
radio (like ath9k)?

Thanks,
Ben


--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com

2020-01-29 19:50:20

by Marlon Smith

[permalink] [raw]
Subject: Re: Strange performance issue when using two devices at once

On Wed, 2020-01-29 at 11:27 -0800, Ben Greear wrote:
> On 1/29/20 11:22 AM, Marlon Smith wrote:
> >
> > On Wed, 2020-01-29 at 10:42 -0800, Ben Greear wrote:
> > >
> > > On 1/29/20 10:39 AM, Marlon Smith wrote:
> > > >
> > > >
> > > > Hi everyone,
> > > >
> > > > I have two RT5370 devices connected to the same access point.
> > > > Both
> > > > devices are very slow, but the instant I disconnect one device,
> > > > the
> > > > other speeds up by a factor of 10.
> > > Out of curiosity, are both of the RT5370 used on the same client
> > > device?
> > > >
> > > > Did you check that they have unique MAC addresses?
> > > > Thanks,
> > > Ben
> > > >
> > > > >
> > > > >
> > > > The really strange part is that one device will perform slowly
> > > > even
> > > > if
> > > > the other device is basically idle! I've confirmed this with a
> > > > packet
> > > > sniffer.
> > > >
> > > > I've been trying to do some debugging, and I've found that when
> > > > both
> > > > devices are connected to the access point, they report a large
> > > > number
> > > > of duplicate frames. I added some debug output
> > > > in ieee80211_rx_h_check_dup() to confirm that this only happens
> > > > while
> > > > both devices are connected. The packet sniffer also shows a
> > > > large
> > > > number of retries while this is occurring.
> > > >
> > > > Using backports 5.3-rc4 for this, but also tested on 4.14-rc2.
> > > >
> > > > I did post about this previously on this mailing list (RT5370
> > > > performance issues), but I thought I'd post again with this new
> > > > information and more descriptive title. I'm a little bit stuck
> > > > on
> > > > this
> > > > for a while now, so any ideas are much appreciated.
> > > >
> > > > Thanks!
> > > >
> > > > Marlon
> > > >
> > They are on separate devices, although the mac addresses are close.
> > 70:F1:1C:2E:AF:B4 and
> > 70:F1:1C:2E:AF:B6.
> >
> > However, I have a third device 70:F1:1C:2E:AF:BB which performs
> > well
> > and does not affect the performance of the other two.
> >
> Have you tried a different AP?
>
> And also tried using the exact same MAC addresses configured on a
> different
> radio (like ath9k)?
>
> Thanks,
> Ben
>
>

I have tried a different access point in a different environment but no
luck. I'll see if I can configure my laptop to use one of the
problematic devices' mac address.

2020-01-29 20:06:21

by Ben Greear

[permalink] [raw]
Subject: Re: Strange performance issue when using two devices at once

On 1/29/20 11:48 AM, Marlon Smith wrote:
> On Wed, 2020-01-29 at 11:27 -0800, Ben Greear wrote:
>> On 1/29/20 11:22 AM, Marlon Smith wrote:
>>>
>>> On Wed, 2020-01-29 at 10:42 -0800, Ben Greear wrote:
>>>>
>>>> On 1/29/20 10:39 AM, Marlon Smith wrote:
>>>>>
>>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I have two RT5370 devices connected to the same access point.
>>>>> Both
>>>>> devices are very slow, but the instant I disconnect one device,
>>>>> the
>>>>> other speeds up by a factor of 10.
>>>> Out of curiosity, are both of the RT5370 used on the same client
>>>> device?
>>>>>
>>>>> Did you check that they have unique MAC addresses?
>>>>> Thanks,
>>>> Ben
>>>>>
>>>>>>
>>>>>>
>>>>> The really strange part is that one device will perform slowly
>>>>> even
>>>>> if
>>>>> the other device is basically idle! I've confirmed this with a
>>>>> packet
>>>>> sniffer.
>>>>>
>>>>> I've been trying to do some debugging, and I've found that when
>>>>> both
>>>>> devices are connected to the access point, they report a large
>>>>> number
>>>>> of duplicate frames. I added some debug output
>>>>> in ieee80211_rx_h_check_dup() to confirm that this only happens
>>>>> while
>>>>> both devices are connected. The packet sniffer also shows a
>>>>> large
>>>>> number of retries while this is occurring.
>>>>>
>>>>> Using backports 5.3-rc4 for this, but also tested on 4.14-rc2.
>>>>>
>>>>> I did post about this previously on this mailing list (RT5370
>>>>> performance issues), but I thought I'd post again with this new
>>>>> information and more descriptive title. I'm a little bit stuck
>>>>> on
>>>>> this
>>>>> for a while now, so any ideas are much appreciated.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Marlon
>>>>>
>>> They are on separate devices, although the mac addresses are close.
>>> 70:F1:1C:2E:AF:B4 and
>>> 70:F1:1C:2E:AF:B6.
>>>
>>> However, I have a third device 70:F1:1C:2E:AF:BB which performs
>>> well
>>> and does not affect the performance of the other two.
>>>
>> Have you tried a different AP?
>>
>> And also tried using the exact same MAC addresses configured on a
>> different
>> radio (like ath9k)?
>>
>> Thanks,
>> Ben
>>
>>
>
> I have tried a different access point in a different environment but no
> luck. I'll see if I can configure my laptop to use one of the
> problematic devices' mac address.

It might be tricky to determine, but if you can notice whether one of your station devices
is (block-)acking the other's frames, that would be a good clue that it is a station
side bug. A carefully inspected sniff, especially if you can put sniffer near one station
and far from the other and so use RSSI as a sorting factor, should allow you to determine
this.

Thanks,
Ben


--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com

2020-01-29 20:50:03

by Marlon Smith

[permalink] [raw]
Subject: Re: Strange performance issue when using two devices at once

On Wed, 2020-01-29 at 12:01 -0800, Ben Greear wrote:
> On 1/29/20 11:48 AM, Marlon Smith wrote:
> >
> > On Wed, 2020-01-29 at 11:27 -0800, Ben Greear wrote:
> > >
> > > On 1/29/20 11:22 AM, Marlon Smith wrote:
> > > >
> > > >
> > > > On Wed, 2020-01-29 at 10:42 -0800, Ben Greear wrote:
> > > > >
> > > > >
> > > > > On 1/29/20 10:39 AM, Marlon Smith wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I have two RT5370 devices connected to the same access
> > > > > > point.
> > > > > > Both
> > > > > > devices are very slow, but the instant I disconnect one
> > > > > > device,
> > > > > > the
> > > > > > other speeds up by a factor of 10.
> > > > > Out of curiosity, are both of the RT5370 used on the same
> > > > > client
> > > > > device?
> > > > > >
> > > > > >
> > > > > > Did you check that they have unique MAC addresses?
> > > > > > Thanks,
> > > > > Ben
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > The really strange part is that one device will perform
> > > > > > slowly
> > > > > > even
> > > > > > if
> > > > > > the other device is basically idle! I've confirmed this
> > > > > > with a
> > > > > > packet
> > > > > > sniffer.
> > > > > >
> > > > > > I've been trying to do some debugging, and I've found that
> > > > > > when
> > > > > > both
> > > > > > devices are connected to the access point, they report a
> > > > > > large
> > > > > > number
> > > > > > of duplicate frames. I added some debug output
> > > > > > in ieee80211_rx_h_check_dup() to confirm that this only
> > > > > > happens
> > > > > > while
> > > > > > both devices are connected. The packet sniffer also shows a
> > > > > > large
> > > > > > number of retries while this is occurring.
> > > > > >
> > > > > > Using backports 5.3-rc4 for this, but also tested on 4.14-
> > > > > > rc2.
> > > > > >
> > > > > > I did post about this previously on this mailing list
> > > > > > (RT5370
> > > > > > performance issues), but I thought I'd post again with this
> > > > > > new
> > > > > > information and more descriptive title. I'm a little bit
> > > > > > stuck
> > > > > > on
> > > > > > this
> > > > > > for a while now, so any ideas are much appreciated.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Marlon
> > > > > >
> > > > They are on separate devices, although the mac addresses are
> > > > close.
> > > > 70:F1:1C:2E:AF:B4 and
> > > > 70:F1:1C:2E:AF:B6.
> > > >
> > > > However, I have a third device 70:F1:1C:2E:AF:BB which performs
> > > > well
> > > > and does not affect the performance of the other two.
> > > >
> > > Have you tried a different AP?
> > >
> > > And also tried using the exact same MAC addresses configured on a
> > > different
> > > radio (like ath9k)?
> > >
> > > Thanks,
> > > Ben
> > >
> > >
> > I have tried a different access point in a different environment
> > but no
> > luck. I'll see if I can configure my laptop to use one of the
> > problematic devices' mac address.
> It might be tricky to determine, but if you can notice whether one of
> your station devices
> is (block-)acking the other's frames, that would be a good clue that
> it is a station
> side bug.  A carefully inspected sniff, especially if you can put
> sniffer near one station
> and far from the other and so use RSSI as a sorting factor, should
> allow you to determine
> this.
>
> Thanks,
> Ben
>
>

I'll double check this, but I believe there is very little traffic from
the second station while the first station is (slowly) transferring
data. I think that means it's not likely that one device is acking the
other's frames?

I will test this though, and the mac address spoofing, and get back to
you tomorrow.

Really appreciate the help so far, thank you!

2020-01-30 14:44:45

by Marlon Smith

[permalink] [raw]
Subject: Re: Strange performance issue when using two devices at once

On Wed, 2020-01-29 at 12:01 -0800, Ben Greear wrote:
> On 1/29/20 11:48 AM, Marlon Smith wrote:
> >
> > On Wed, 2020-01-29 at 11:27 -0800, Ben Greear wrote:
> > >
> > > On 1/29/20 11:22 AM, Marlon Smith wrote:
> > > >
> > > >
> > > > On Wed, 2020-01-29 at 10:42 -0800, Ben Greear wrote:
> > > > >
> > > > >
> > > > > On 1/29/20 10:39 AM, Marlon Smith wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I have two RT5370 devices connected to the same access
> > > > > > point.
> > > > > > Both
> > > > > > devices are very slow, but the instant I disconnect one
> > > > > > device,
> > > > > > the
> > > > > > other speeds up by a factor of 10.
> > > > > Out of curiosity, are both of the RT5370 used on the same
> > > > > client
> > > > > device?
> > > > > >
> > > > > >
> > > > > > Did you check that they have unique MAC addresses?
> > > > > > Thanks,
> > > > > Ben
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > The really strange part is that one device will perform
> > > > > > slowly
> > > > > > even
> > > > > > if
> > > > > > the other device is basically idle! I've confirmed this
> > > > > > with a
> > > > > > packet
> > > > > > sniffer.
> > > > > >
> > > > > > I've been trying to do some debugging, and I've found that
> > > > > > when
> > > > > > both
> > > > > > devices are connected to the access point, they report a
> > > > > > large
> > > > > > number
> > > > > > of duplicate frames. I added some debug output
> > > > > > in ieee80211_rx_h_check_dup() to confirm that this only
> > > > > > happens
> > > > > > while
> > > > > > both devices are connected. The packet sniffer also shows a
> > > > > > large
> > > > > > number of retries while this is occurring.
> > > > > >
> > > > > > Using backports 5.3-rc4 for this, but also tested on 4.14-
> > > > > > rc2.
> > > > > >
> > > > > > I did post about this previously on this mailing list
> > > > > > (RT5370
> > > > > > performance issues), but I thought I'd post again with this
> > > > > > new
> > > > > > information and more descriptive title. I'm a little bit
> > > > > > stuck
> > > > > > on
> > > > > > this
> > > > > > for a while now, so any ideas are much appreciated.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Marlon
> > > > > >
> > > > They are on separate devices, although the mac addresses are
> > > > close.
> > > > 70:F1:1C:2E:AF:B4 and
> > > > 70:F1:1C:2E:AF:B6.
> > > >
> > > > However, I have a third device 70:F1:1C:2E:AF:BB which performs
> > > > well
> > > > and does not affect the performance of the other two.
> > > >
> > > Have you tried a different AP?
> > >
> > > And also tried using the exact same MAC addresses configured on a
> > > different
> > > radio (like ath9k)?
> > >
> > > Thanks,
> > > Ben
> > >
> > >
> > I have tried a different access point in a different environment
> > but no
> > luck. I'll see if I can configure my laptop to use one of the
> > problematic devices' mac address.
> It might be tricky to determine, but if you can notice whether one of
> your station devices
> is (block-)acking the other's frames, that would be a good clue that
> it is a station
> side bug.  A carefully inspected sniff, especially if you can put
> sniffer near one station
> and far from the other and so use RSSI as a sorting factor, should
> allow you to determine
> this.
>
> Thanks,
> Ben
>
>

So I did a few tests and got some pretty interesting results.

With the two devices connected to the access point and performing
slowly, I changed the MAC address of one of the devices (Just from a B6
to a BA). Immediately, both devices began working properly!

Back to the original MAC address, I looked at my sniffer, and I do not
see any evidence of one device acking another device's frames. With one
device sitting idle and the other transferring data, I see high numbers
of retransmitted frames, and then an eventual ack from the correct
device, but very little activity from the idle device. While this is
happening, the device driver indicates that it is dropping lots of
duplicate received packets.

2020-01-30 14:56:18

by Jean-Pierre TOSONI

[permalink] [raw]
Subject: RE: Strange performance issue when using two devices at once

Maybe you need to put some distance between the 3 devices.
We experienced this kind of problem on radios close to each other (around 10cm).
It was due to radio signals inducing themselves in the PCB of another
nearby radio, causing the other radio to badly transmit and receive.

> -----Message d'origine-----
> De : [email protected] [mailto:[email protected]] De la part de
> Marlon Smith
> Envoyé : mercredi 29 janvier 2020 19:40
> À : [email protected]
> Objet : Strange performance issue when using two devices at once
>
> Hi everyone,
>
> I have two RT5370 devices connected to the same access point. Both
> devices are very slow, but the instant I disconnect one device, the
> other speeds up by a factor of 10.
>
> The really strange part is that one device will perform slowly even if
> the other device is basically idle! I've confirmed this with a packet
> sniffer.
>
> I've been trying to do some debugging, and I've found that when both
> devices are connected to the access point, they report a large number
> of duplicate frames. I added some debug output
> in ieee80211_rx_h_check_dup() to confirm that this only happens while
> both devices are connected. The packet sniffer also shows a large
> number of retries while this is occurring.
>
> Using backports 5.3-rc4 for this, but also tested on 4.14-rc2.
>
> I did post about this previously on this mailing list (RT5370
> performance issues), but I thought I'd post again with this new
> information and more descriptive title. I'm a little bit stuck on this
> for a while now, so any ideas are much appreciated.
>
> Thanks!
>
> Marlon

2020-01-30 15:27:39

by Marlon Smith

[permalink] [raw]
Subject: Re: Strange performance issue when using two devices at once

On Thu, 2020-01-30 at 14:55 +0000, Jean-Pierre TOSONI wrote:
> Maybe you need to put some distance between the 3 devices.
> We experienced this kind of problem on radios close to each other
> (around 10cm).
> It was due to radio signals inducing themselves in the PCB of another
> nearby radio, causing the other radio to badly transmit and receive.
>
> >
> > -----Message d'origine-----
> > De : [email protected] [mailto:linux-wireless-ow
> > [email protected]] De la part de
> > Marlon Smith
> > Envoyé : mercredi 29 janvier 2020 19:40
> > À : [email protected]
> > Objet : Strange performance issue when using two devices at once
> >
> > Hi everyone,
> >
> > I have two RT5370 devices connected to the same access point. Both
> > devices are very slow, but the instant I disconnect one device, the
> > other speeds up by a factor of 10.
> >
> > The really strange part is that one device will perform slowly even
> > if
> > the other device is basically idle! I've confirmed this with a
> > packet
> > sniffer.
> >
> > I've been trying to do some debugging, and I've found that when
> > both
> > devices are connected to the access point, they report a large
> > number
> > of duplicate frames. I added some debug output
> > in ieee80211_rx_h_check_dup() to confirm that this only happens
> > while
> > both devices are connected. The packet sniffer also shows a large
> > number of retries while this is occurring.
> >
> > Using backports 5.3-rc4 for this, but also tested on 4.14-rc2.
> >
> > I did post about this previously on this mailing list (RT5370
> > performance issues), but I thought I'd post again with this new
> > information and more descriptive title. I'm a little bit stuck on
> > this
> > for a while now, so any ideas are much appreciated.
> >
> > Thanks!
> >
> > Marlon

Thanks for the reply Jean-Pierre. I just discovered this morning that
if I change the MAC address of one of the devices, they both perform
well. So I'm wondering if it is a driver issue.

However I will keep this in mind, as I often do have multiple devices
together in close proximity.

2020-01-30 15:33:40

by Felix Fietkau

[permalink] [raw]
Subject: Re: Strange performance issue when using two devices at once

On 2020-01-30 15:43, Marlon Smith wrote:
> So I did a few tests and got some pretty interesting results.
>
> With the two devices connected to the access point and performing
> slowly, I changed the MAC address of one of the devices (Just from a B6
> to a BA). Immediately, both devices began working properly!
>
> Back to the original MAC address, I looked at my sniffer, and I do not
> see any evidence of one device acking another device's frames. With one
> device sitting idle and the other transferring data, I see high numbers
> of retransmitted frames, and then an eventual ack from the correct
> device, but very little activity from the idle device. While this is
> happening, the device driver indicates that it is dropping lots of
> duplicate received packets.
Here's the cause and an approach for fixing it:

The MAC_BSSID_DW1 register configures the Multi-BSSID mode. It can be
configured for 1, 2, 4 or 8 BSSIDs. This controls what MAC addresses the
chip will respond to. The driver always sets it up for 8 BSSIDs.

The problem here is that the chip has a hardcoded pattern for how those
BSSIDs are allocated:
Depending on the number of BSSIDs, the lower 1, 2 or 3 bits are used to
encode the BSS index. This means that when 8 BSSIDs are enabled, the
lower 3 bits are effectively masked out on the MAC address filter that
decides which frames to ACK. Because your MAC addresses are so close
together, both devices try to ACK the same frames at the same time. This
will not be visible in a sniffer, since they will be transmitting at the
same time.

I'd say the first step in fixing this is to make the driver set
MAC_BSSID_DW1_BSS_ID_MASK based on the MAC addresses of configured
virtual interfaces. For older chips, this MAC address pattern cannot be
fixed.

However, on at least RT3883, RT3352 and RT5350 (and thus most likely
also RT5370), you can set BIT 21 in that register, which changes the
pattern. With that bit set, the second interface only sets the local bit
of the MAC address, and from the third interface on it adds (idx-1)<<2
to the first byte. That gets rid of the MAC address conflicts.

Hopefully someone familiar with the rt2x00 code will read this and write
some patches :)

- Felix

2020-01-30 16:05:16

by Marlon Smith

[permalink] [raw]
Subject: Re: Strange performance issue when using two devices at once

On Thu, 2020-01-30 at 16:32 +0100, Felix Fietkau wrote:
> On 2020-01-30 15:43, Marlon Smith wrote:
> >
> > So I did a few tests and got some pretty interesting results.
> >
> > With the two devices connected to the access point and performing
> > slowly, I changed the MAC address of one of the devices (Just from
> > a B6
> > to a BA). Immediately, both devices began working properly!
> >
> > Back to the original MAC address, I looked at my sniffer, and I do
> > not
> > see any evidence of one device acking another device's frames. With
> > one
> > device sitting idle and the other transferring data, I see high
> > numbers
> > of retransmitted frames, and then an eventual ack from the correct
> > device, but very little activity from the idle device. While this
> > is
> > happening, the device driver indicates that it is dropping lots of
> > duplicate received packets.
> Here's the cause and an approach for fixing it:
>
> The MAC_BSSID_DW1 register configures the Multi-BSSID mode. It can be
> configured for 1, 2, 4 or 8 BSSIDs. This controls what MAC addresses
> the
> chip will respond to. The driver always sets it up for 8 BSSIDs.
>
> The problem here is that the chip has a hardcoded pattern for how
> those
> BSSIDs are allocated:
> Depending on the number of BSSIDs, the lower 1, 2 or 3 bits are used
> to
> encode the BSS index. This means that when 8 BSSIDs are enabled, the
> lower 3 bits are effectively masked out on the MAC address filter
> that
> decides which frames to ACK. Because your MAC addresses are so close
> together, both devices try to ACK the same frames at the same time.
> This
> will not be visible in a sniffer, since they will be transmitting at
> the
> same time.
>
> I'd say the first step in fixing this is to make the driver set
> MAC_BSSID_DW1_BSS_ID_MASK based on the MAC addresses of configured
> virtual interfaces. For older chips, this MAC address pattern cannot
> be
> fixed.
>
> However, on at least RT3883, RT3352 and RT5350 (and thus most likely
> also RT5370), you can set BIT 21 in that register, which changes the
> pattern. With that bit set, the second interface only sets the local
> bit
> of the MAC address, and from the third interface on it adds (idx-
> 1)<<2
> to the first byte. That gets rid of the MAC address conflicts.
>
> Hopefully someone familiar with the rt2x00 code will read this and
> write
> some patches :)
>
> - Felix


Felix, this was exactly it. I set that bit and the problem disappeared.
Thank you so much!

Ben, your help was very much appreciated as well. Your suggestions
really helped narrow things down.

Is there a way I can help others who might come across this problem in
the future? Would filing a bug report help anyone out, even if just to
document this issue?

Thanks again both of you, you saved me countless hours of time here.

Marlon