Hi,
I'd discussed this briefly yesterday with Eliad but think I should write
it down more coherently (as if :) ) and also for wider dissemination.
We're already running into problems with remain-on-channel where they
want to flush the non-off-channel frames, but can only stop the queues
globally which then also stops off-channel frames. We haven't seen these
issues yet in our device but I think we probably will see it too.
This will be exacerbated by multi-channel, so I thought we probably need
to actually resolve this problem. Currently, the only indication the
driver has that a frame is off-channel is IEEE80211_TX_CTL_TX_OFFCHAN,
which only our driver seems to use. However, even that isn't sufficient,
if the off-channel period happens to fall into a period where the
regular queues are stopped, mac80211 will not send out the frame to
start with.
As a consequence, I'm thinking that we should redesign the mac80211 /
driver queue API to mirror more closely the kind of queues hardware
really has.
To recap: today, almost all drivers expose four queues to mac80211 for
the four ACs. In reality, many have many more queues, e.g. in iwlwifi:
* 4 for the first vif
* 4 for the second vif
* 1 for CAB (content after [DTIM] beacon)
* 1 for off-channel
They are mapped as follows:
* all vif ACs -> 4 ACs
* off-channel -> BE
* CAB -> BE (I believe, maybe all?)
Thus when e.g. in our driver the BE queue for the second VIF is full, we
stop all traffic across all VIFs. When both VIFs are on the same
channel, this isn't really a problem. However, when they are on
different channels, there could be vastly different performance
characteristics on those two channels due to interference etc. Currently
we also don't even handle the CAB or off-channel queue getting full
(which doesn't really happen, but it actually could, particularly CAB)
As a result of all of this, I think it may make sense to redesign the
queue API to better mirror what's going on. I'm thinking that the driver
should expose all of the queues and indicate to mac80211 how to sort
frames into them.
So what changes would this require?
First of all, of course the driver needs to advertise the number of
hardware queues it supports. We'll have to think about changing the
check "do we support QoS" from "hw.queues >= 4" to something else, but I
think >=4 will still be a good indication and it seems unlikely that
somebody will have a device with a lot of queues yet not support QoS.
Then of course, the driver needs to tell mac80211 about queue mapping.
For this, there's one global queue:
* off-channel TX queue
Additionally, there are per-interface queues:
* 4 per AC
* CAB for this interface
The biggest change would be that we now have to modify the queue mapping
at the driver/mac80211 API boundary. Incoming frames from the network
layer would still get their queue mapping to be set to the AC (since
each virtual interface keeps the four AC software queues) but then to
the driver we have to set the queue mapping to the right hardware queue.
We might decide to use another field in the tx control data, but we have
no space in there :-)
Now when a driver stops a queue, we would stop it in mac80211, and then
instead of going through all interfaces and stopping the corresponding
queue there we'd go through all interfaces and check if it's used for
that interface and then stop the corresponding queue in the netdev.
(Ditto for wake, obviously.)
The way I see this being used is having four queues per concurrent
channel that the device supports, plus a few of the auxiliary queues.
This is necessary anyway for quick channel switching (you can't flush to
switch channels, too slow). Then each virtual interface gets its queues
mapped into the right hardware queues, and the driver actually has less
work to do in re-mapping the AC (we do that today.)
Thoughts? Is that description coherent enough? :-)
johannes
This is a plot of applying queue management on top of the existing
queue structure, only one queue active, against 50 saturating streams
against a voip-like ping.
http://www.teklibre.com/~d/bloat/ping_log.ps
To explain it the first part of the plot is the start of the test,
the second part is after a brief, small rate change downwards, the
drop to 0 is the end of the first test, then it goes back and starts
another.
While the clustering at the ~60-90ms line is good and due to sfqred
doing it's job, it's connected to and unable to control the large,
floppy rubber hose of buffering beneath it, translating out to ~100ms
of jitter on more normal spiky loads under good conditions and worse
on bad.
A flaw of this particular plot is it measures intrinsic buffering on
both sides of the connection, but it's still a lot of intrinsic
latency. It would be nice to be able to apply BQL or BQL-like
techniques to this, because that white space underneath the plot gets
much more latent at lower rates.
On Mon, Feb 27, 2012 at 4:10 PM, Dave Taht <[email protected]> wrote:
> On Mon, Feb 27, 2012 at 12:49 PM, Johannes Berg
> <[email protected]> wrote:
>
>> As a consequence, I'm thinking that we should redesign the mac80211 /
>> driver queue API to mirror more closely the kind of queues hardware
>> really has.
>
> I'd like the driver queues to mirror more closely the kinds of
> connections the hardware really has. I realize that this is more of an
> AP orientation than a wireless client orientation, but with wildly
> different rates between connected devices bad things happen.
>
> Secondly, my take on 802.11e QoS and wireless-n aggregation is that
> aggregation wins nearly every time; all
> 802.11e does is bloat up the buffers.
>
>> To recap: today, almost all drivers expose four queues to mac80211 for
>> the four ACs. In reality, many have many more queues, e.g. in iwlwifi:
>> ?* 4 for the first vif
>> ?* 4 for the second vif
>> ?* 1 for CAB (content after [DTIM] beacon)
>> ?* 1 for off-channel
>>
>> They are mapped as follows:
>> ?* all vif ACs -> 4 ACs
>> ?* off-channel -> BE
>> ?* CAB ? ? ? ? -> BE (I believe, maybe all?)
>>
>> Thus when e.g. in our driver the BE queue for the second VIF is full, we
>> stop all traffic across all VIFs. When both VIFs are on the same
>> channel, this isn't really a problem. However, when they are on
>> different channels, there could be vastly different performance
>> characteristics on those two channels due to interference etc.
>
> And even more differences based on the destination's characteristics.
>
>
>
>
> --
> Dave T?ht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> http://www.bufferbloat.net
--
Dave T?ht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net
On Mon, Feb 27, 2012 at 4:46 PM, Johannes Berg
<[email protected]> wrote:
> On Mon, 2012-02-27 at 16:38 +0100, Helmut Schaa wrote:
>> On Mon, Feb 27, 2012 at 12:49 PM, Johannes Berg
>> <[email protected]> wrote:
>> > Then of course, the driver needs to tell mac80211 about queue mapping.
>> > For this, there's one global queue:
>> > ?* off-channel TX queue
>> > Additionally, there are per-interface queues:
>> > ?* 4 per AC
>> > ?* CAB for this interface
>>
>> Ralink devices have only the 4 hw queues and no CAB queue. So, would
>> each interface point to the same hw queue or would the driver have to do
>> some sort of translation?
>
> No, just like today I'd expect each interface to point to the same hw
> queue. In fact, even in devices that do have more queues I would expect
> multiple interfaces to point to the same hw queue, if they share
> channels.
>
> This will be somewhat tricky when an the mapping isn't static, but for
> that the netdev queues can be stopped and then the reassignment is
> probably not so much of a problem.
Ok.
>> Also Ralink devices don't have a CAB queue :)
>
> You just said that ;-)
>
> But in that case I suppose the CAB queue would simply point to the BE
> queue or so as well. But I think they are also quite broken wrt. sending
> multicast-after-DTIM anyway, no?
Yep, that's done in a tasklet using the BE queue at the moment. Not the
perfect solution ...
In any case I'd then expect this to
> either point to a "virtual" queue that the driver simply has in
> software, or to the real BE queue, depending on the desired queue-full
> semantics.
Ok, fine with me.
Helmut
On Mon, Feb 27, 2012 at 12:49 PM, Johannes Berg
<[email protected]> wrote:
> Then of course, the driver needs to tell mac80211 about queue mapping.
> For this, there's one global queue:
> ?* off-channel TX queue
> Additionally, there are per-interface queues:
> ?* 4 per AC
> ?* CAB for this interface
Ralink devices have only the 4 hw queues and no CAB queue. So, would
each interface point to the same hw queue or would the driver have to do
some sort of translation?
Also Ralink devices don't have a CAB queue :)
Helmut
On Mon, 2012-02-27 at 16:10 +0100, Dave Taht wrote:
> On Mon, Feb 27, 2012 at 12:49 PM, Johannes Berg
> <[email protected]> wrote:
>
> > As a consequence, I'm thinking that we should redesign the mac80211 /
> > driver queue API to mirror more closely the kind of queues hardware
> > really has.
>
> I'd like the driver queues to mirror more closely the kinds of
> connections the hardware really has. I realize that this is more of an
> AP orientation than a wireless client orientation, but with wildly
> different rates between connected devices bad things happen.
This isn't possible -- having 4 netdev queues per remote station is,
simply put, insane. We've discussed this before on this list and Eric
had a few suggestions, but I have no time to work on them. Essentially
though it boils down to solving the problem at a higher layer by
differentiating traffic to different stations.
> Secondly, my take on 802.11e QoS and wireless-n aggregation is that
> aggregation wins nearly every time; all
> 802.11e does is bloat up the buffers.
That statement doesn't really make sense to me -- QoS (on the air) and
aggregation are two unrelated things?
> > Thus when e.g. in our driver the BE queue for the second VIF is full, we
> > stop all traffic across all VIFs. When both VIFs are on the same
> > channel, this isn't really a problem. However, when they are on
> > different channels, there could be vastly different performance
> > characteristics on those two channels due to interference etc.
>
> And even more differences based on the destination's characteristics.
I do understand that this is true, but we don't have the ability to
solve this problem at the netdev queue level.
johannes
On Mon, Feb 27, 2012 at 12:49 PM, Johannes Berg
<[email protected]> wrote:
> As a consequence, I'm thinking that we should redesign the mac80211 /
> driver queue API to mirror more closely the kind of queues hardware
> really has.
I'd like the driver queues to mirror more closely the kinds of
connections the hardware really has. I realize that this is more of an
AP orientation than a wireless client orientation, but with wildly
different rates between connected devices bad things happen.
Secondly, my take on 802.11e QoS and wireless-n aggregation is that
aggregation wins nearly every time; all
802.11e does is bloat up the buffers.
> To recap: today, almost all drivers expose four queues to mac80211 for
> the four ACs. In reality, many have many more queues, e.g. in iwlwifi:
> ?* 4 for the first vif
> ?* 4 for the second vif
> ?* 1 for CAB (content after [DTIM] beacon)
> ?* 1 for off-channel
>
> They are mapped as follows:
> ?* all vif ACs -> 4 ACs
> ?* off-channel -> BE
> ?* CAB ? ? ? ? -> BE (I believe, maybe all?)
>
> Thus when e.g. in our driver the BE queue for the second VIF is full, we
> stop all traffic across all VIFs. When both VIFs are on the same
> channel, this isn't really a problem. However, when they are on
> different channels, there could be vastly different performance
> characteristics on those two channels due to interference etc.
And even more differences based on the destination's characteristics.
--
Dave T?ht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net
On Mon, 2012-02-27 at 16:38 +0100, Helmut Schaa wrote:
> On Mon, Feb 27, 2012 at 12:49 PM, Johannes Berg
> <[email protected]> wrote:
> > Then of course, the driver needs to tell mac80211 about queue mapping.
> > For this, there's one global queue:
> > * off-channel TX queue
> > Additionally, there are per-interface queues:
> > * 4 per AC
> > * CAB for this interface
>
> Ralink devices have only the 4 hw queues and no CAB queue. So, would
> each interface point to the same hw queue or would the driver have to do
> some sort of translation?
No, just like today I'd expect each interface to point to the same hw
queue. In fact, even in devices that do have more queues I would expect
multiple interfaces to point to the same hw queue, if they share
channels.
This will be somewhat tricky when an the mapping isn't static, but for
that the netdev queues can be stopped and then the reassignment is
probably not so much of a problem.
> Also Ralink devices don't have a CAB queue :)
You just said that ;-)
But in that case I suppose the CAB queue would simply point to the BE
queue or so as well. But I think they are also quite broken wrt. sending
multicast-after-DTIM anyway, no? In any case I'd then expect this to
either point to a "virtual" queue that the driver simply has in
software, or to the real BE queue, depending on the desired queue-full
semantics.
johannes
On 03/26/12 12:13, Johannes Berg wrote:
> So I've started to work on this a bit.
>
> On Mon, 2012-02-27 at 12:49 +0100, Johannes Berg wrote:
>
>> So what changes would this require?
>>
>> First of all, of course the driver needs to advertise the number of
>> hardware queues it supports. We'll have to think about changing the
>> check "do we support QoS" from "hw.queues >= 4" to something else, but I
>> think >=4 will still be a good indication and it seems unlikely that
>> somebody will have a device with a lot of queues yet not support QoS.
> This is a non-issue really.
>
>> Then of course, the driver needs to tell mac80211 about queue mapping.
>> For this, there's one global queue:
>> * off-channel TX queue
>> Additionally, there are per-interface queues:
>> * 4 per AC
>> * CAB for this interface
> This I haven't gotten to.
>
> One issue I noticed though is that the queue mapping might need to be
> changed during the lifetime of the virtual interface (ie. while it's
> UP).
>
> Over in the other thread, I'm discussing channel contexts with Michal,
> and it seems that when the channel context is changed the queue mapping
> might also change. This doesn't seem like a problem, but we definitely
> need to keep it in mind. The way to handle it would be to stop the
> netdev queues, sync_net(), drop everything we have queued up for this
> interface and then we have clean state to start from.
>
>
> The other thing this affects is monitor mode. When the hardware queues
> are only defined per interface, we can't just give a frame to the driver
> on an essentially random queue and hope it can transmit it. Therefore,
> we need more queues, for monitor mode, in some way.
>
> As I was also trying to figure out what a good way to handle monitor
> mode in cases like iwlwifi is where the device really wants to be in a
> special "monitor mode" -- not just different filter programming.
>
> So to address both of these things, I came up with this patch:
> http://p.sipsolutions.net/ad87702d716df6d5.txt
>
> This will allow drivers to be informed of monitor mode when that is the
> only thing active.
>
> Injection would work like this:
> * when monitor is only active:
> - use this pseudo monitor interface
> - if that doesn't exist (driver doesn't have it): discard frame
> * when something else is active:
> - if the TA of the injected frame matches any vif: use that vif
> - if the TA doesn't match: use a vif matching the monitor addr
> - if that doesn't exist: discard frame
>
>
> I think this strikes a good balance between having injection and having
> to worry about it too much. Obviously, drivers not implementing this new
> monitor mode interface capability wouldn't have pure injection, but that
> seems OK -- it should be relatively easy to add this to drivers.
This all seems very sane to me, I don't have any complaints to register.
> As mentioned in the patch, the alternative would be to have new API for
> all of this, but that doesn't seem worthwhile.
Agreed
thanks,
Rick
>
> johannes
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
So I've started to work on this a bit.
On Mon, 2012-02-27 at 12:49 +0100, Johannes Berg wrote:
> So what changes would this require?
>
> First of all, of course the driver needs to advertise the number of
> hardware queues it supports. We'll have to think about changing the
> check "do we support QoS" from "hw.queues >= 4" to something else, but I
> think >=4 will still be a good indication and it seems unlikely that
> somebody will have a device with a lot of queues yet not support QoS.
This is a non-issue really.
> Then of course, the driver needs to tell mac80211 about queue mapping.
> For this, there's one global queue:
> * off-channel TX queue
> Additionally, there are per-interface queues:
> * 4 per AC
> * CAB for this interface
This I haven't gotten to.
One issue I noticed though is that the queue mapping might need to be
changed during the lifetime of the virtual interface (ie. while it's
UP).
Over in the other thread, I'm discussing channel contexts with Michal,
and it seems that when the channel context is changed the queue mapping
might also change. This doesn't seem like a problem, but we definitely
need to keep it in mind. The way to handle it would be to stop the
netdev queues, sync_net(), drop everything we have queued up for this
interface and then we have clean state to start from.
The other thing this affects is monitor mode. When the hardware queues
are only defined per interface, we can't just give a frame to the driver
on an essentially random queue and hope it can transmit it. Therefore,
we need more queues, for monitor mode, in some way.
As I was also trying to figure out what a good way to handle monitor
mode in cases like iwlwifi is where the device really wants to be in a
special "monitor mode" -- not just different filter programming.
So to address both of these things, I came up with this patch:
http://p.sipsolutions.net/ad87702d716df6d5.txt
This will allow drivers to be informed of monitor mode when that is the
only thing active.
Injection would work like this:
* when monitor is only active:
- use this pseudo monitor interface
- if that doesn't exist (driver doesn't have it): discard frame
* when something else is active:
- if the TA of the injected frame matches any vif: use that vif
- if the TA doesn't match: use a vif matching the monitor addr
- if that doesn't exist: discard frame
I think this strikes a good balance between having injection and having
to worry about it too much. Obviously, drivers not implementing this new
monitor mode interface capability wouldn't have pure injection, but that
seems OK -- it should be relatively easy to add this to drivers.
As mentioned in the patch, the alternative would be to have new API for
all of this, but that doesn't seem worthwhile.
johannes