2012-03-20 14:40:02

by Michal Kazior

[permalink] [raw]
Subject: [RFC 00/12] multi-channel support

Hello,

The following patches prepare mac80211 to support multi-channel capable
hardware. The patchset prepares to channel per-vif split.

Work still needs to be done:
* powersave per-vif
* queue locking per-vif
* offchannel rework (hw_config, work_work)
* and a bit more

Questions:

* monitor interfaces:
Currently ieee80211_set_channel gets netdev==NULL when iface is
a monitor. Is there a particular reason behind it?

* ieee80211_hw_config:
Should we extend it to take ieee80211_sub_if_data or should we
use ieee80211_bss_info_change_notify? If so, is ieee80211_hw_config
eventually to be removed?

What do you think of this approach?


-- Pozdrawiam / Best regards, Michal Kazior.


Michal Kazior (12):
mac80211: prepare ieee80211_mandatory_rates to per-vif
mac80211: prepare ieee80211_sta_get_rates to per-vif
mac80211: prepare ieee80211_frame_duration to per-vif
mac80211: prepare ieee80211_get_tx_channel_type to per-vif
mac80211: split channel state
mac80211: move offchannel flag
mac80211: change conf_is_ht* to take ieee80211_channel_conf
mac80211: introduce IEEE80211_HW_SUPPORTS_MULTI_CHANNEL
mac80211: add initial structures for multi-channel
mac80211: rename ieee80211_offchannel_return
mac80211: split offchannel functions to per-vif
mac80211: return NULL from get_channel in multi-channel

drivers/net/wireless/adm8211.c | 2 +-
drivers/net/wireless/at76c50x-usb.c | 4 +-
drivers/net/wireless/ath/ath5k/base.c | 2 +-
drivers/net/wireless/ath/ath5k/mac80211-ops.c | 4 +-
drivers/net/wireless/ath/ath9k/ani.c | 8 +-
drivers/net/wireless/ath/ath9k/beacon.c | 2 +-
drivers/net/wireless/ath/ath9k/calib.c | 6 +-
drivers/net/wireless/ath/ath9k/common.c | 4 +-
drivers/net/wireless/ath/ath9k/debug.c | 4 +-
drivers/net/wireless/ath/ath9k/htc_drv_main.c | 26 ++--
drivers/net/wireless/ath/ath9k/htc_drv_txrx.c | 12 +-
drivers/net/wireless/ath/ath9k/hw.c | 8 +-
drivers/net/wireless/ath/ath9k/main.c | 14 +-
drivers/net/wireless/ath/ath9k/rc.c | 18 ++--
drivers/net/wireless/ath/ath9k/recv.c | 8 +-
drivers/net/wireless/ath/carl9170/debug.c | 4 +-
drivers/net/wireless/ath/carl9170/mac.c | 12 +-
drivers/net/wireless/ath/carl9170/main.c | 6 +-
drivers/net/wireless/ath/carl9170/phy.c | 12 +-
drivers/net/wireless/b43/b43.h | 2 +-
drivers/net/wireless/b43/main.c | 12 +-
drivers/net/wireless/b43/phy_ht.c | 4 +-
drivers/net/wireless/b43/phy_lcn.c | 4 +-
drivers/net/wireless/b43/phy_n.c | 4 +-
drivers/net/wireless/b43legacy/main.c | 8 +-
.../net/wireless/brcm80211/brcmsmac/mac80211_if.c | 6 +-
drivers/net/wireless/iwlegacy/3945-rs.c | 2 +-
drivers/net/wireless/iwlegacy/4965-mac.c | 6 +-
drivers/net/wireless/iwlegacy/4965-rs.c | 8 +-
drivers/net/wireless/iwlegacy/common.c | 10 +-
drivers/net/wireless/iwlwifi/iwl-agn-rs.c | 10 +-
drivers/net/wireless/iwlwifi/iwl-agn-rxon.c | 10 +-
drivers/net/wireless/iwlwifi/iwl-mac80211.c | 2 +-
drivers/net/wireless/libertas_tf/main.c | 8 +-
drivers/net/wireless/mac80211_hwsim.c | 8 +-
drivers/net/wireless/mwl8k.c | 30 +++---
drivers/net/wireless/p54/fwio.c | 4 +-
drivers/net/wireless/p54/main.c | 4 +-
drivers/net/wireless/p54/txrx.c | 4 +-
drivers/net/wireless/rt2x00/rt2800lib.c | 26 ++--
drivers/net/wireless/rt2x00/rt2x00config.c | 16 ++--
drivers/net/wireless/rt2x00/rt61pci.c | 2 +-
drivers/net/wireless/rt2x00/rt73usb.c | 2 +-
drivers/net/wireless/rtl818x/rtl8180/dev.c | 4 +-
drivers/net/wireless/rtl818x/rtl8180/grf5101.c | 2 +-
drivers/net/wireless/rtl818x/rtl8180/max2820.c | 2 +-
drivers/net/wireless/rtl818x/rtl8180/rtl8225.c | 2 +-
drivers/net/wireless/rtl818x/rtl8180/sa2400.c | 2 +-
drivers/net/wireless/rtl818x/rtl8187/dev.c | 4 +-
drivers/net/wireless/rtl818x/rtl8187/rtl8225.c | 2 +-
drivers/net/wireless/rtlwifi/base.c | 4 +-
drivers/net/wireless/rtlwifi/core.c | 6 +-
drivers/net/wireless/rtlwifi/rtl8192ce/trx.c | 4 +-
drivers/net/wireless/rtlwifi/rtl8192cu/trx.c | 8 +-
drivers/net/wireless/rtlwifi/rtl8192de/trx.c | 4 +-
drivers/net/wireless/rtlwifi/rtl8192se/trx.c | 4 +-
drivers/net/wireless/wl1251/main.c | 4 +-
drivers/net/wireless/wl12xx/main.c | 14 +-
drivers/net/wireless/zd1211rw/zd_mac.c | 4 +-
include/net/mac80211.h | 41 ++++--
net/mac80211/cfg.c | 57 ++++++----
net/mac80211/chan.c | 12 ++-
net/mac80211/debugfs.c | 5 +-
net/mac80211/driver-trace.h | 4 +-
net/mac80211/ibss.c | 32 +++--
net/mac80211/ieee80211_i.h | 54 +++++++--
net/mac80211/iface.c | 24 +++-
net/mac80211/main.c | 50 ++++----
net/mac80211/mesh.c | 20 ++--
net/mac80211/mesh_hwmp.c | 3 +-
net/mac80211/mesh_plink.c | 10 +-
net/mac80211/mlme.c | 57 ++++++----
net/mac80211/offchannel.c | 126 +++++++++++---------
net/mac80211/rate.h | 3 +-
net/mac80211/rc80211_minstrel.c | 12 +-
net/mac80211/rc80211_minstrel_ht.c | 13 ++-
net/mac80211/rx.c | 5 +-
net/mac80211/scan.c | 10 +-
net/mac80211/tx.c | 19 ++-
net/mac80211/util.c | 67 +++++++----
net/mac80211/work.c | 23 ++--
81 files changed, 608 insertions(+), 457 deletions(-)



2012-03-27 15:00:29

by Johannes Berg

[permalink] [raw]
Subject: Re: [RFC 00/12] multi-channel support

On Tue, 2012-03-27 at 16:41 +0200, Michal Kazior wrote:
> Johannes Berg wrote:
> > Ok, so ... I think we're getting to the bottom of our misunderstanding&
> > confusion. I've completely ignored scan/off-channel in this because I
> > don't have to handle it in software, our device handles it itself. It
> > seems from this that you do need it handled in software.
>
> No, not really. I was just worried as to not break things and maybe
> improve them as a bonus.

Ah, ok. I agree we should be careful to not break things, but I don't
think we should do more than we have to. In particular, I think adding
multi-channel software implementation is too much :-)


> > Now let's go back to the question of off-channel/scan -- this seems to
> > be the fundamental difference in how we look at things. First, the same
> > thing applies here. SW scanning in mac80211 right now is very
> > inefficient when you try to do background scanning, since it doesn't
> > synchronise with beacons etc. With multiple virtual interfaces, this
> > will only get worse. With multiple channels, probably more so.
> >
> > Then also I'm in a bit of a luxurious position -- our device implements
> > scanning and off-channel all by itself. For this reason, but also
> > because of the timing, I've always wanted to ignore off-channel and
> > scanning in the multi-channel code as much as possible.
> >
> > Your proposal to add temporary channel contexts might be feasible, but I
> > wonder how complex it would become and how much of it can really be
> > shared? I expect that ath9k will implement multi-channel completely in
> > software, and any scheduler there would probably also roll in
> > scanning/off-channel. (Not that I know, I don't work for QCA :) )
>
> It would be nice if someone from Atheros camp could give us a hint on that.

Yeah, though I'm not sure I care at this stage? I'm sure when (if?) they
implement this they can come up with a proposal, and if it ends up being
useful for somebody else at some point then they can work together to
design the code to handle it all.


> > Would that seem okay to you? I'd rather leave this complexity out of
> > mac80211, not just for complexity reasons but also to make the API have
> > easier semantics.
>
> I get the general idea but I can't imagine any detailed examples. I'm
> yet to fully understand mac80211 and other drivers/devices.

Ok. Basically I'm punting software-off-channel/-scan in multi-channel
context -- is that something you could work with for your device? Does
your device have hw-scan/hw-off-channel?


johannes


2012-03-26 12:27:50

by Michal Kazior

[permalink] [raw]
Subject: Re: [RFC 00/12] multi-channel support

Hi,

Sorry for my late reply.


Johannes Berg wrote:
> On Fri, 2012-03-23 at 15:03 +0100, Michał Kazior wrote:
>>> So I took a closer look at this now.
>>>
>>> I think we should probably apply the first two patches now, and the
>>> third with changes. After that, I'm not so sure.
>>
>> You mean the following, right?
>>
>> mac80211: prepare ieee80211_mandatory_rates to per-vif
>> mac80211: prepare ieee80211_sta_get_rates to per-vif
>> [tofix] mac80211: prepare ieee80211_frame_duration to per-vif
>>
>> Should I resend them all, or just the 3rd one once I fix it?
>
> You should probably send them as [PATCH].

Okay.


>> What if we create a new structure that would be a channel
>> stream/reservation? Driver could then report multiple such objects. Each
>> such would have a tx queue and it's own locking. Let's assume something
>> like:
>>
>> struct ieee80211_channel_stream {
>> struct list_head vif; // list of vifs we're attached to
>> struct ieee80211_channel *ch;
>> struct sk_buff_head pending;
>> int stop_reasons[IEEE80211_MAX_QUEUES];
>> // .. other stuff
>> u8 *drv_priv[0] __attribute__((__aligned__((void *))));
>> };
>>
>> struct ieee80211_vif {
>> struct ieee80211_channel_stream *channel_stream;
>> // .. rest of struct
>> };
>>
>> * We could then reject set_channel() based on number of different
>> oper_channels and channel reservations
>> * We could handle off-channel through a channel reservation by either
>> using an unused one (which would result in hardware offloaded
>> off-channel) or we would lock&flush it, switch to off-channel, do
>> work, and get back
>
> I'm not sure we can easily use such a "stream" for off-channel
> operation. In particular, that "stream" might not support multiple
> queues.

Does it need to? It could as well just map all queues to a single hw
queue with this design. I'm not sure if I get your point here.


Is it okay for us to not care if the driver supports real queues for
ACs? Should we care if a driver maps all ACs from a channel context (or
vif for that matter) onto a single hw queue? I'm wondering whether that
would simplify code paths?


>> We should however consider that a device may not support certain
>> interface combinations. One thing that comes immediately to my mind is
>> the following: suppose we support STA+STA on different channels, AP+AP
>> on the same channel, but do not support AP+AP on different channels:
>>
>> 1. create wlan0, set channel=1, and start AP
>> 2. create wlan1, set channel=1, and start AP
>> 3. do set_channel(wlan1, 3)<-- oops
>>
>> This would probably be fixed easily with:
>>
>> 1. forbid changing channels on a running interface (i'm aware that's
>> the same case for mac addr)
>> 2. thus we would require: ifdown + setchannel + ifup
>>
>> But that's just one case. Are there any others? This needs investigation.
>
> So my interface combinations advertising already supports
> "num_different_channels" per combination, so that would be easy to
> express. Switching the channel while the AP interface is up is probably
> not desirable anyway.

You're right. I've looked at the combination structures and they seem
quite nice.


>
>> What is still missing here is the CAB queue (and possible other that may
>> come up in the future) you mentioned in the other thread.
>
> Ah, I see you did see my other thread :-)
>
>> I'm not sure
>> whether we may use queue_mapping for our own evil purposes here. If not,
>> then how about this:
>>
>> struct ieee80211_queue_types { // or tx_queue_types?
>> IEEE80211_QUEUE_VO,
>> IEEE80211_QUEUE_VI,
>> IEEE80211_QUEUE_BE,
>> IEEE80211_QUEUE_BK,
>> IEEE80211_QUEUE_CAB,
>>
>> IEEE80211_QUEUE_LAST
>> };
> [...]
>> This looks kind of ugly, but oh well. This way the driver will be able
>> to setup mapping between mac80211 and internal device queues. It should
>> also be possible to map one hardware queue to multiple
>> ieee80211_channel_stream structures (queues is a list of pointers) so
>> multiple vifs on the same channel is okay.
>
> I don't think this makes a lot of sense, tbh. This isn't a separate AC,
> I'd rather keep them separated.

You mean the CAB queue separated from ACs?


>
>> What are your thoughts on this? Is this terribly wrong or is it the way
>> to go? Have I missed something?
>
> I think that this is all a bit overkill :)
>
> I'm looking at the code right now, and I think we should treat ACs,
> queues and channels separately. If we combine queue sets& channels into
> one, I think we'll find drivers that don't work correctly with this.
> Some devices for instance might have a set of queues for each interface,
> even if they're on the same channel. I don't want to lose flexibility to
> handle that.

You mean a driver that can handle two vifs simultaneously but on the
same channel and have two separate queue sets for that?

Then we need to move queues to ieee80211_vif and create callbacks to
bind channels contexts with a vif (just you mentioned that in other email).


>
> I think we'd hold the AC in skb_queue_mapping, since that's assigned
> early in the virtual interface queues already, and we need it to be the
> AC there for proper queue selection.
>
> The hardware queue we can put into a separate field in tx_info, I've
> made space for a u8 for this in a patch I have pending (but not sent to
> the list yet.)
>
> Ignoring the monitor mode interface, I think the simplest thing we can
> do is
>
> struct ieee80211_hw {
> ...
> u8 offchannel_queue;
> };
>
> struct ieee80211_vif {
> ...
> u8 hw_queue[IEEE80211_NUM_ACS];
> u8 after_dtim_queue;
> };
>
> For any of these (offchannel_queue, after_dtim_queue, hw_queue[i]), the
> queue number Q must be 0<= Q< hw.queues. This allows us to keep all
> per-hw-queue state "flattened" in ieee80211_local's pending,
> queue_stop_reasons, etc. instead of separating all of it out into
> separate structures.

So you want "offchannel_queue", "hw_queue[]" and "after_dtim_queue" to
point to driver/device hw queue id?


>
> When a queue is stopped, we do:
>
> void queue_stop(int q)
> {
> for_each_interface(vif) {
> if (vif->hw_queue[AC] == q) // for all 4 ACs
> stop netdev queue (vif->dev, AC)
> if (vif->after_dtim_queue == q)
> stop_all_queues(vif->dev)
> }
> }
>
> Or so ... after_dtim_queue is kinda a special case.

Oh, I see. So stopping CAB would be like "stop all the things".

But how do we then check if we need to stop given queue or not? Let's
say a driver stops q=2 which corresponds to BE on vif0 and vif1. But
then comes mac80211 aggregation and stops BE on vif0. I can only think
of a single solution to that: "u8 lock_reasons[256];" in ieee80211_local
but it seems like an overkill or is it not?

>
>
> If offchannel_queue is stopped, we'll have to refuse offchannel TX from
> nl80211 -- that seems *very* unlikely though.
>
>
> Do you think this would work?
>
> I think that covers the queue part -- I'll split the thread and will
> talk about multi-channel in a separate email.
>
>
> johannes
>


-- Pozdrawiam / Best regards, Michal Kazior.


2012-03-26 12:29:25

by Michal Kazior

[permalink] [raw]
Subject: Re: [RFC 00/12] multi-channel support

Johannes Berg wrote:
> On Fri, 2012-03-23 at 15:03 +0100, Michał Kazior wrote:
>
>> What if we create a new structure that would be a channel
>> stream/reservation? Driver could then report multiple such objects.
>> Each such would have a tx queue and it's own locking. Let's assume
>> something like:
>>
>> struct ieee80211_channel_stream {
>> struct list_head vif; // list of vifs we're attached to
>> struct ieee80211_channel *ch;
>> struct sk_buff_head pending;
>> int stop_reasons[IEEE80211_MAX_QUEUES];
>> // .. other stuff
>> u8 *drv_priv[0] __attribute__((__aligned__((void *))));
>> };
>
> I'm not sure I see the need to have the driver register these, but it
> seems useful to maintain this abstraction somehow in mac80211.
>
> Due to the interface combinations advertising, we already know (*) what
> is supported by the driver, so there's no need for the driver to
> register this again.
>
> Like I said earlier, I see basically two ways of handling this.
>
> The explicit way would essentially consist of something like this:
>
> struct ieee80211_channel_context {
> struct ieee80211_channel *chan;
> enum nl80211_channel_type type;
> };
>
> with API to add/remove such contexts from/to the device:
>
> struct ieee80211_ops {
> ...
> void (*add_channel_context)(hw, ctx),
> void (*remove_channel_context)(hw, ctx),
> void (*change_channel_type)(hw, ctx),
> void (*assign_vif_channel_context)(hw, vif, ctx),
> };
>
> The assign function would have rather strange semantics though -- it can
> be called only when the interface is up (added to the driver), but it
> can't be called while the interface is associated, active as an AP, etc.
> EXCEPT ... maybe it can in CSA scenarios. But I said that before: CSA is
> a bit of a mess to handle.

Are we going to assume that "add_channel_context" always succeeds?
So I understand you want to use interface combinations to control when
these functions are valid to be called.

But can we express the "off-channel only" channel context through
interface combinations? Is it valid for a driver to say the following?

struct ieee80211_iface_limit limits[] = {
{ .max = 1, .types = BIT(NL80211_IFTYPE_AP) },
{ .max = 1, .types = BIT(NL80211_IFTYPE_STATION) },
{ .max = 1, .types = 0 }
};

struct ieee80211_iface_combination combination = {
.limits = limits,
.n_limits = 3,
.max_interfaces = 2,
.num_different_channels = 3,
};


You also assume that a driver can do a generic "add_channel_context"
that would allow it to prepare the device. This may not be true for all
devices. It is unknown whether this channel context is going to be used
for off-channel, STA or AP. Especially off-channel is broken this way
since it isn't bound to vif - in fact it can't be. The same goes for
CSA. Or maybe I didn't get the idea behind "add_channel_context"?


How about:

struct ieee80211_ops {
// ..
void (*changed_channel_type)(hw, ctx);
void (*detach_channel_context)(ctx);
void (*attach_channel_context_to_vif)(hw, vif, ctx);
void (*attach_channel_context_to_hw)(hw, ctx);
};

This way mac80211 immediately states it's intention - and we can use
"attach_channel_context_to_hw" to do off-channel, and
"attach_channel_context_to_vif" to do CSA as well as normal interface
start up. We could maybe split detach_* into two different functions if
necessary. The "attach_channel_context_to_vif" would be allowed even
when associated for the purpose of dealing with CSA.

What do you think?


The off-channel flow would be then:

* hw supports 1 interface on 1 channel
* we're connected as STA
1. scan request
2. stop sta
3. detach_channel_context(oper) without freeing it of course
4. create channel context "tmp"
5. set next scan channel in channel context "tmp"
6. attach_channel_context_to_hw(tmp)
7. work
8. detach_channel_context(tmp)
9. go to 5 if more work needed
10. detach_channel_context(tmp)
11. attach_channel_context(oper)

Are we okay with this?

Also if the device were to support at least 1 extra channel that can do
(offloaded) off-channel we wouldn't have to do steps: 2, 3, 11.


> Now that I think about it, the implicit way I previously thought was
> quite a separate mechanism is really very similar: you'd still have to
> have the assign_vif_channel_context function, maybe as part of
> bss_info_changed, but not have explicit add/remove. I think the
> add/remove is useful so let's not consider implicit methods any more :)

I'm unsure about the add/remove. See above. We can't just add a channel
context without stating the purpose of that channel context. A driver
might need to know of the purpose beforehand.


> The internal handling in mac80211 might have a linked list of the
> channel contexts, or just a static array of a handful of them, doesn't
> really matter. It would, however, make sure that there aren't more than
> the current interface combination advertised as supported (see (*)
> again.)
>
> Of course mac80211 would also assign the interfaces to channel contexts
> as needed -- if a given interface is already on channel 1, the next one
> wouldn't need a new channel context if it also uses channel 1 and the
> channel types are compatible. But that reminds me -- when we do that we
> may have to update the channel context to change its channel type, so
> add an operation for that.
>
>
> Wrt. off-channel, I'm not sure I've understood you correctly. The
> current off-channel is a very short-lived thing that doesn't really use
> a "complete" channel context. In fact, in our implementation it is
> completely separate since it's temporary (**).

Maybe you're right, but how can we then express off-channel in a neat
clean way? If we use channel contexts we get software AND hardware
off-channel (as long as the device supports it) at the same time.

If it's a concern that userspace may want to start a new interface on a
different channel while off-channel is taking up "a precious channel
context", we may either do -EBUSY or just sleep a few seconds/defer the
work if the off-channel is really short-lived.

Also if we consider channel contexts that can't do normal operation at
all - there wouldn't be such problem since they would never be used for
anything but off-channel.


>
> As far as scanning is concerned, I'm not really sure. I haven't spent
> much time thinking about it since we have hardware offload for it (and
> also off-channel) -- I'm not sure what a good solution would be.
>
> johannes
>
>
> (*) after we refactor cfg80211_can_change_interface() and export
> cfg80211_get_current_combination() that returns the pointer to the
> combination that's currently in use, or something like that. We'd have
> to figure out if another combination is available that has more channels
> though, if needed.

Is there a lot of work to be done?


> (**) However, we're thinking about implementing a P2P-device context
> that would allow offloading things like periodic listen
> (discoverability) or find (search + listen). That would probably use a
> channel context then.

But that wouldn't need a permanent channel context, would it? It'll
probably be cheap to attach/detach channel contexts.


>> Should we add capability
>> flags for each ieee80211_channel_stream? Or we could add new driver
>> callback, eg. "set_channel_try" and allow the driver to decide
>> whether it allows a certain hardware state or not.
>
> This is not a fully feasible "stream" anyway. Think of it more like a
> single scan dwell -- you wouldn't express that as a separate "stream"
> I think?

Why not? Should we care what vif we're working with, or hw for that
matter? Off-channel, as well as other operation means we want to do
something on a certain channel. It shouldn't matter how long we use that
channel (except if we're borrowing time from another operational mode).


Another question would be if we could do software multi-channel
operation. Do you think it's even possible and worth exploring?



-- Pozdrawiam / Best regards, Michal Kazior.


2012-03-23 09:29:07

by Johannes Berg

[permalink] [raw]
Subject: Re: [RFC 00/12] multi-channel support

Hi,

> The following patches prepare mac80211 to support multi-channel capable
> hardware. The patchset prepares to channel per-vif split.
>
> Work still needs to be done:
> * powersave per-vif
> * queue locking per-vif
> * offchannel rework (hw_config, work_work)
> * and a bit more
>
> Questions:
>
> * monitor interfaces:
> Currently ieee80211_set_channel gets netdev==NULL when iface is
> a monitor. Is there a particular reason behind it?
>
> * ieee80211_hw_config:
> Should we extend it to take ieee80211_sub_if_data or should we
> use ieee80211_bss_info_change_notify? If so, is ieee80211_hw_config
> eventually to be removed?
>
> What do you think of this approach?


So I took a closer look at this now.

I think we should probably apply the first two patches now, and the
third with changes. After that, I'm not so sure.

Overall, I think the approach isn't sufficient. At the very least, the
(unstated!) assumption that one channel per interface can always be done
is bad. Modifying this with the structure you gave it though seems to be
problematic.

We should also handle the queue mapping first I think. This isn't a
feature you can implement overnight :-)

johannes



2012-03-23 14:31:16

by Johannes Berg

[permalink] [raw]
Subject: Re: [RFC 00/12] multi-channel support

On Fri, 2012-03-23 at 15:03 +0100, Michał Kazior wrote:
> > So I took a closer look at this now.
> >
> > I think we should probably apply the first two patches now, and the
> > third with changes. After that, I'm not so sure.
>
> You mean the following, right?
>
> mac80211: prepare ieee80211_mandatory_rates to per-vif
> mac80211: prepare ieee80211_sta_get_rates to per-vif
> [tofix] mac80211: prepare ieee80211_frame_duration to per-vif
>
> Should I resend them all, or just the 3rd one once I fix it?

You should probably send them as [PATCH].


> > Overall, I think the approach isn't sufficient. At the very least, the
> > (unstated!) assumption that one channel per interface can always be done
> > is bad. Modifying this with the structure you gave it though seems to be
> > problematic.
> >
> > We should also handle the queue mapping first I think. This isn't a
> > feature you can implement overnight :-)
>
> I've been thinking more on queue handling in multi-channel. Here are my
> thoughts.

Did you see my previous mail?
http://mid.gmane.org/[email protected]


> If we assume that off-channel should be a global thing then we should
> rethink the whole queue thing first.

I think we need to rethink the queues first even if off-channel isn't
global -- right now you can only stop a given AC across all interfaces
but that's really not what you want if you're on multiple channels...


> What if we create a new structure that would be a channel
> stream/reservation? Driver could then report multiple such objects. Each
> such would have a tx queue and it's own locking. Let's assume something
> like:
>
> struct ieee80211_channel_stream {
> struct list_head vif; // list of vifs we're attached to
> struct ieee80211_channel *ch;
> struct sk_buff_head pending;
> int stop_reasons[IEEE80211_MAX_QUEUES];
> // .. other stuff
> u8 *drv_priv[0] __attribute__((__aligned__((void *))));
> };
>
> struct ieee80211_vif {
> struct ieee80211_channel_stream *channel_stream;
> // .. rest of struct
> };
>
> * We could then reject set_channel() based on number of different
> oper_channels and channel reservations
> * We could handle off-channel through a channel reservation by either
> using an unused one (which would result in hardware offloaded
> off-channel) or we would lock&flush it, switch to off-channel, do
> work, and get back

I'm not sure we can easily use such a "stream" for off-channel
operation. In particular, that "stream" might not support multiple
queues.

> We should however consider that a device may not support certain
> interface combinations. One thing that comes immediately to my mind is
> the following: suppose we support STA+STA on different channels, AP+AP
> on the same channel, but do not support AP+AP on different channels:
>
> 1. create wlan0, set channel=1, and start AP
> 2. create wlan1, set channel=1, and start AP
> 3. do set_channel(wlan1, 3) <-- oops
>
> This would probably be fixed easily with:
>
> 1. forbid changing channels on a running interface (i'm aware that's
> the same case for mac addr)
> 2. thus we would require: ifdown + setchannel + ifup
>
> But that's just one case. Are there any others? This needs investigation.

So my interface combinations advertising already supports
"num_different_channels" per combination, so that would be easy to
express. Switching the channel while the AP interface is up is probably
not desirable anyway.

The biggest challenge here is CSA functionality I think.


> Other issue I'm thinking would be: what if a device supports specialized
> off-channel queue (is there one we know? iwlwifi?) ?

Yeah we do have this.

> We can't advertise
> this as a fully-usable ieee80211_channel_stream (again, is there
> anything that has a shortcoming like that?).

Yup.

> Should we add capability
> flags for each ieee80211_channel_stream? Or we could add new driver
> callback, eg. "set_channel_try" and allow the driver to decide whether
> it allows a certain hardware state or not.

This is not a fully feasible "stream" anyway. Think of it more like a
single scan dwell -- you wouldn't express that as a separate "stream" I
think?

> What is still missing here is the CAB queue (and possible other that may
> come up in the future) you mentioned in the other thread.

Ah, I see you did see my other thread :-)

> I'm not sure
> whether we may use queue_mapping for our own evil purposes here. If not,
> then how about this:
>
> struct ieee80211_queue_types { // or tx_queue_types?
> IEEE80211_QUEUE_VO,
> IEEE80211_QUEUE_VI,
> IEEE80211_QUEUE_BE,
> IEEE80211_QUEUE_BK,
> IEEE80211_QUEUE_CAB,
>
> IEEE80211_QUEUE_LAST
> };
[...]
> This looks kind of ugly, but oh well. This way the driver will be able
> to setup mapping between mac80211 and internal device queues. It should
> also be possible to map one hardware queue to multiple
> ieee80211_channel_stream structures (queues is a list of pointers) so
> multiple vifs on the same channel is okay.

I don't think this makes a lot of sense, tbh. This isn't a separate AC,
I'd rather keep them separated.

> What are your thoughts on this? Is this terribly wrong or is it the way
> to go? Have I missed something?

I think that this is all a bit overkill :)

I'm looking at the code right now, and I think we should treat ACs,
queues and channels separately. If we combine queue sets & channels into
one, I think we'll find drivers that don't work correctly with this.
Some devices for instance might have a set of queues for each interface,
even if they're on the same channel. I don't want to lose flexibility to
handle that.

I think we'd hold the AC in skb_queue_mapping, since that's assigned
early in the virtual interface queues already, and we need it to be the
AC there for proper queue selection.

The hardware queue we can put into a separate field in tx_info, I've
made space for a u8 for this in a patch I have pending (but not sent to
the list yet.)

Ignoring the monitor mode interface, I think the simplest thing we can
do is

struct ieee80211_hw {
...
u8 offchannel_queue;
};

struct ieee80211_vif {
...
u8 hw_queue[IEEE80211_NUM_ACS];
u8 after_dtim_queue;
};

For any of these (offchannel_queue, after_dtim_queue, hw_queue[i]), the
queue number Q must be 0 <= Q < hw.queues. This allows us to keep all
per-hw-queue state "flattened" in ieee80211_local's pending,
queue_stop_reasons, etc. instead of separating all of it out into
separate structures.

When a queue is stopped, we do:

void queue_stop(int q)
{
for_each_interface(vif) {
if (vif->hw_queue[AC] == q) // for all 4 ACs
stop netdev queue (vif->dev, AC)
if (vif->after_dtim_queue == q)
stop_all_queues(vif->dev)
}
}

Or so ... after_dtim_queue is kinda a special case.


If offchannel_queue is stopped, we'll have to refuse offchannel TX from
nl80211 -- that seems *very* unlikely though.


Do you think this would work?

I think that covers the queue part -- I'll split the thread and will
talk about multi-channel in a separate email.


johannes


2012-03-23 14:03:20

by Michal Kazior

[permalink] [raw]
Subject: Re: [RFC 00/12] multi-channel support

Johannes Berg wrote:
> Hi,
>
>> The following patches prepare mac80211 to support multi-channel capable
>> hardware. The patchset prepares to channel per-vif split.
>>
>> Work still needs to be done:
>> * powersave per-vif
>> * queue locking per-vif
>> * offchannel rework (hw_config, work_work)
>> * and a bit more
>>
>> Questions:
>>
>> * monitor interfaces:
>> Currently ieee80211_set_channel gets netdev==NULL when iface is
>> a monitor. Is there a particular reason behind it?
>>
>> * ieee80211_hw_config:
>> Should we extend it to take ieee80211_sub_if_data or should we
>> use ieee80211_bss_info_change_notify? If so, is ieee80211_hw_config
>> eventually to be removed?
>>
>> What do you think of this approach?
>
>
> So I took a closer look at this now.
>
> I think we should probably apply the first two patches now, and the
> third with changes. After that, I'm not so sure.

You mean the following, right?

mac80211: prepare ieee80211_mandatory_rates to per-vif
mac80211: prepare ieee80211_sta_get_rates to per-vif
[tofix] mac80211: prepare ieee80211_frame_duration to per-vif

Should I resend them all, or just the 3rd one once I fix it?


> Overall, I think the approach isn't sufficient. At the very least, the
> (unstated!) assumption that one channel per interface can always be done
> is bad. Modifying this with the structure you gave it though seems to be
> problematic.
>
> We should also handle the queue mapping first I think. This isn't a
> feature you can implement overnight :-)

I've been thinking more on queue handling in multi-channel. Here are my
thoughts.

If we assume that off-channel should be a global thing then we should
rethink the whole queue thing first.

What if we create a new structure that would be a channel
stream/reservation? Driver could then report multiple such objects. Each
such would have a tx queue and it's own locking. Let's assume something
like:

struct ieee80211_channel_stream {
struct list_head vif; // list of vifs we're attached to
struct ieee80211_channel *ch;
struct sk_buff_head pending;
int stop_reasons[IEEE80211_MAX_QUEUES];
// .. other stuff
u8 *drv_priv[0] __attribute__((__aligned__((void *))));
};

struct ieee80211_vif {
struct ieee80211_channel_stream *channel_stream;
// .. rest of struct
};

* We could then reject set_channel() based on number of different
oper_channels and channel reservations
* We could handle off-channel through a channel reservation by either
using an unused one (which would result in hardware offloaded
off-channel) or we would lock&flush it, switch to off-channel, do
work, and get back

We should however consider that a device may not support certain
interface combinations. One thing that comes immediately to my mind is
the following: suppose we support STA+STA on different channels, AP+AP
on the same channel, but do not support AP+AP on different channels:

1. create wlan0, set channel=1, and start AP
2. create wlan1, set channel=1, and start AP
3. do set_channel(wlan1, 3) <-- oops

This would probably be fixed easily with:

1. forbid changing channels on a running interface (i'm aware that's
the same case for mac addr)
2. thus we would require: ifdown + setchannel + ifup

But that's just one case. Are there any others? This needs investigation.

Other issue I'm thinking would be: what if a device supports specialized
off-channel queue (is there one we know? iwlwifi?) ? We can't advertise
this as a fully-usable ieee80211_channel_stream (again, is there
anything that has a shortcoming like that?). Should we add capability
flags for each ieee80211_channel_stream? Or we could add new driver
callback, eg. "set_channel_try" and allow the driver to decide whether
it allows a certain hardware state or not.

What is still missing here is the CAB queue (and possible other that may
come up in the future) you mentioned in the other thread. I'm not sure
whether we may use queue_mapping for our own evil purposes here. If not,
then how about this:

struct ieee80211_queue_types { // or tx_queue_types?
IEEE80211_QUEUE_VO,
IEEE80211_QUEUE_VI,
IEEE80211_QUEUE_BE,
IEEE80211_QUEUE_BK,
IEEE80211_QUEUE_CAB,

IEEE80211_QUEUE_LAST
};

enum ieee80211_queue_types
ieee80211_ac_number_to_queue_type(enum ieee80211_ac_numbers ac);

struct ieee80211_channel_stream_queue {
enum ieee80211_queue_types type;
u8 drv_priv[0];
};

struct XYZ_channel_stream_queue { // example driver priv
int internal_queue_id;
// other stuff, like driver queues, stats, etc.
};

struct ieee80211_channel_stream {
struct ieee80211_channel_stream_queue *queues[IEEE80211_QUEUE_LAST];
// ..
};

struct ieee80211_vif_queue {
struct ieee80211_vif *vif;
struct ieee80211_channel_stream_queue *queue;
};

struct ieee80211_vif {
// this is going to be just a helper
// we can't refer to ieee80211_channel_stream_queue directly since
// it wouldn't know what vif it belongs to (it may belong to
// multiple vifs)
struct ieee80211_vif_queue queues[IEEE80211_QUEUE_LAST];
// ..
};

struct ieee80211_tx_info {
// instead of ieee80211_vif* we use ieee80211_vif_queue*
};

This looks kind of ugly, but oh well. This way the driver will be able
to setup mapping between mac80211 and internal device queues. It should
also be possible to map one hardware queue to multiple
ieee80211_channel_stream structures (queues is a list of pointers) so
multiple vifs on the same channel is okay.


What are your thoughts on this? Is this terribly wrong or is it the way
to go? Have I missed something?


-- Pozdrawiam / Best Regards, Michal Kazior.

2012-03-27 11:42:41

by Michal Kazior

[permalink] [raw]
Subject: Re: [RFC 00/12] multi-channel support

Johannes Berg wrote:
> Hi,
>
>>> I'm not sure we can easily use such a "stream" for off-channel
>>> operation. In particular, that "stream" might not support multiple
>>> queues.
>>
>> Does it need to? It could as well just map all queues to a single hw
>> queue with this design. I'm not sure if I get your point here.
>>
>>
>> Is it okay for us to not care if the driver supports real queues for
>> ACs? Should we care if a driver maps all ACs from a channel context (or
>> vif for that matter) onto a single hw queue? I'm wondering whether that
>> would simplify code paths?
>
> Well, ok, that would be possible, but I'm not sure I see the point in
> using a stream for off-channel. Today, we completely offload off-channel
> to the device, and I don't see how that could change? Having temporary
> context allocations all the time seems like a lot of thrash?

We could pre-allocate a fixed number of channel contexts based on the
interface combinations, so we'd be "only" left with rebinding those
contexts (and queue mappings along too).

I guess this part of the discussion is moot since we're going to put
queues in vifs (and not in channel contexts).


>> But how do we then check if we need to stop given queue or not? Let's
>> say a driver stops q=2 which corresponds to BE on vif0 and vif1. But
>> then comes mac80211 aggregation and stops BE on vif0. I can only think
>> of a single solution to that: "u8 lock_reasons[256];" in ieee80211_local
>> but it seems like an overkill or is it not?
>
> Essentially that's what I was considering, we'd maintain all the queue
> state per HW queue ID. I said it would be a u8 for the HW queue ID, but
> I don't see anyone using more than say 32 or so, so we wouldn't really
> need 256. But we could go up to 256 if needed (though at that point we
> should probably do dynamic allocation instead.)

We could maybe allocate it according to hw.queues? I think it should be
okay unless a driver has some kind of non-static queues. Are you aware
of such a device/driver?



-- Pozdrawiam / Best regards, Michal Kazior.


2012-03-27 14:41:06

by Michal Kazior

[permalink] [raw]
Subject: Re: [RFC 00/12] multi-channel support

Johannes Berg wrote:
> Ok, so ... I think we're getting to the bottom of our misunderstanding&
> confusion. I've completely ignored scan/off-channel in this because I
> don't have to handle it in software, our device handles it itself. It
> seems from this that you do need it handled in software.

No, not really. I was just worried as to not break things and maybe
improve them as a bonus.


>>> Wrt. off-channel, I'm not sure I've understood you correctly. The
>>> current off-channel is a very short-lived thing that doesn't really use
>>> a "complete" channel context. In fact, in our implementation it is
>>> completely separate since it's temporary (**).
>>
>> Maybe you're right, but how can we then express off-channel in a neat
>> clean way? If we use channel contexts we get software AND hardware
>> off-channel (as long as the device supports it) at the same time.
>
> I'm not sure. The detach logic etc. you proposed above will really not
> be implementable in our device's context handling.

Right. I was trying to compensate for the sw off-channel channel
contexts. I think I'm just going overboard and forget about practical
usage of things sometimes, silly me.


>> >> Should we add capability
>> >> flags for each ieee80211_channel_stream? Or we could add new driver
>> >> callback, eg. "set_channel_try" and allow the driver to decide
>> >> whether it allows a certain hardware state or not.
>> >
>> > This is not a fully feasible "stream" anyway. Think of it more like a
>> > single scan dwell -- you wouldn't express that as a separate "stream"
>> > I think?
>>
>> Why not? Should we care what vif we're working with, or hw for that
>> matter? Off-channel, as well as other operation means we want to do
>> something on a certain channel. It shouldn't matter how long we use that
>> channel (except if we're borrowing time from another operational mode).
>
> I'm not sure how your device works, but it seems that the device would
> require binding vifs to a channel context so that it knows which is
> active where? When to synchronise with beacons, for example?

I was thinking in general "what if".


>> Another question would be if we could do software multi-channel
>> operation. Do you think it's even possible and worth exploring?
>
> I don't think it's feasible to implement in mac80211 because of timing
> constraints. With devices like ath9k it can probably be done by relying
> on hardware timers, interrupts and queue blocking/opening, but because
> mac80211 has to deal with USB and SDIO chips too which can be really
> slow it can't really do this. So no, I wouldn't think it's worth
> exploring at least not directly in mac80211 -- maybe as a separate
> helper library or something.

I see. But then again - one should not expect software multi-channel to
yield great performance, in fact I would expect it to be very slow.


> Now let's go back to the question of off-channel/scan -- this seems to
> be the fundamental difference in how we look at things. First, the same
> thing applies here. SW scanning in mac80211 right now is very
> inefficient when you try to do background scanning, since it doesn't
> synchronise with beacons etc. With multiple virtual interfaces, this
> will only get worse. With multiple channels, probably more so.
>
> Then also I'm in a bit of a luxurious position -- our device implements
> scanning and off-channel all by itself. For this reason, but also
> because of the timing, I've always wanted to ignore off-channel and
> scanning in the multi-channel code as much as possible.
>
> Your proposal to add temporary channel contexts might be feasible, but I
> wonder how complex it would become and how much of it can really be
> shared? I expect that ath9k will implement multi-channel completely in
> software, and any scheduler there would probably also roll in
> scanning/off-channel. (Not that I know, I don't work for QCA :) )

It would be nice if someone from Atheros camp could give us a hint on that.


> Also, I'm a bit afraid that doing so would make the channel context API
> even more complex, and it's not really easy anyway.
>
> I think I need to know more about how you would need to use those
> temporary channel contexts, but I have a feeling that we might be better
> off by splitting it into a separate helper library. I'm thinking this
> would essentially sit between mac80211 and the driver, and offer
> "translation services", i.e. it would have a "remain_on_channel"
> function that the driver could call for mac80211's remain_on_channel
> callback, and then this separate code could do something like your
> temporary channel context detach/attach etc.
>
> Would that seem okay to you? I'd rather leave this complexity out of
> mac80211, not just for complexity reasons but also to make the API have
> easier semantics.

I get the general idea but I can't imagine any detailed examples. I'm
yet to fully understand mac80211 and other drivers/devices.


--
Pozdrawiam / Best regards, Michal Kazior.

2012-03-26 13:19:54

by Johannes Berg

[permalink] [raw]
Subject: Re: [RFC 00/12] multi-channel support

On Mon, 2012-03-26 at 14:29 +0200, Michal Kazior wrote:
> Johannes Berg wrote:
> > On Fri, 2012-03-23 at 15:03 +0100, Michał Kazior wrote:
> >
> >> What if we create a new structure that would be a channel
> >> stream/reservation? Driver could then report multiple such objects.
> >> Each such would have a tx queue and it's own locking. Let's assume
> >> something like:
> >>
> >> struct ieee80211_channel_stream {
> >> struct list_head vif; // list of vifs we're attached to
> >> struct ieee80211_channel *ch;
> >> struct sk_buff_head pending;
> >> int stop_reasons[IEEE80211_MAX_QUEUES];
> >> // .. other stuff
> >> u8 *drv_priv[0] __attribute__((__aligned__((void *))));
> >> };
> >
> > I'm not sure I see the need to have the driver register these, but it
> > seems useful to maintain this abstraction somehow in mac80211.
> >
> > Due to the interface combinations advertising, we already know (*) what
> > is supported by the driver, so there's no need for the driver to
> > register this again.
> >
> > Like I said earlier, I see basically two ways of handling this.
> >
> > The explicit way would essentially consist of something like this:
> >
> > struct ieee80211_channel_context {
> > struct ieee80211_channel *chan;
> > enum nl80211_channel_type type;
> > };
> >
> > with API to add/remove such contexts from/to the device:
> >
> > struct ieee80211_ops {
> > ...
> > void (*add_channel_context)(hw, ctx),
> > void (*remove_channel_context)(hw, ctx),
> > void (*change_channel_type)(hw, ctx),
> > void (*assign_vif_channel_context)(hw, vif, ctx),
> > };
> >
> > The assign function would have rather strange semantics though -- it can
> > be called only when the interface is up (added to the driver), but it
> > can't be called while the interface is associated, active as an AP, etc.
> > EXCEPT ... maybe it can in CSA scenarios. But I said that before: CSA is
> > a bit of a mess to handle.
>
> Are we going to assume that "add_channel_context" always succeeds?
> So I understand you want to use interface combinations to control when
> these functions are valid to be called.

I think we have to assume that. The driver is able to express its
restrictions through the interface combinations, and hopefully it can
live up to those expectations?


> But can we express the "off-channel only" channel context through
> interface combinations? Is it valid for a driver to say the following?
>
> struct ieee80211_iface_limit limits[] = {
> { .max = 1, .types = BIT(NL80211_IFTYPE_AP) },
> { .max = 1, .types = BIT(NL80211_IFTYPE_STATION) },
> { .max = 1, .types = 0 }
> };
>
> struct ieee80211_iface_combination combination = {
> .limits = limits,
> .n_limits = 3,
> .max_interfaces = 2,
> .num_different_channels = 3,
> };

This doesn't make much sense, the third limit entry would never be used
since the off-channel code doesn't consider channel contexts here.

> You also assume that a driver can do a generic "add_channel_context"
> that would allow it to prepare the device. This may not be true for all
> devices. It is unknown whether this channel context is going to be used
> for off-channel, STA or AP. Especially off-channel is broken this way
> since it isn't bound to vif - in fact it can't be. The same goes for
> CSA. Or maybe I didn't get the idea behind "add_channel_context"?

I'm thinking that the channel context would always be bound to a vif. If
the driver didn't need this to prepare the device, then it would simply
do the entire device setup when the context is bound to the vif? Or even
later when the vif is in use?


> How about:
>
> struct ieee80211_ops {
> // ..
> void (*changed_channel_type)(hw, ctx);
> void (*detach_channel_context)(ctx);
> void (*attach_channel_context_to_vif)(hw, vif, ctx);
> void (*attach_channel_context_to_hw)(hw, ctx);
> };
>
> This way mac80211 immediately states it's intention - and we can use
> "attach_channel_context_to_hw" to do off-channel, and
> "attach_channel_context_to_vif" to do CSA as well as normal interface
> start up. We could maybe split detach_* into two different functions if
> necessary. The "attach_channel_context_to_vif" would be allowed even
> when associated for the purpose of dealing with CSA.
>
> What do you think?

I'm not sure I understand your desire to handle off-channel within this
framework. An off-channel period is, by definition, very time-limited.
As such, I don't see why it matters whether you do:

ctx = add_channel_context(off-channel channel)
attach_channel_context_to_hw(hw, ctx)
remain_on_channel(hw, ctx, duration)
delete_channel_context(ctx)

or this:
remain_on_channel(hw, channel, duration)

In the latter case, the driver can set up and tear down the context
internally just as well, since it's only temporary?


> The off-channel flow would be then:
>
> * hw supports 1 interface on 1 channel
> * we're connected as STA
> 1. scan request
> 2. stop sta
> 3. detach_channel_context(oper) without freeing it of course
> 4. create channel context "tmp"
> 5. set next scan channel in channel context "tmp"
> 6. attach_channel_context_to_hw(tmp)
> 7. work
> 8. detach_channel_context(tmp)
> 9. go to 5 if more work needed
> 10. detach_channel_context(tmp)
> 11. attach_channel_context(oper)
>
> Are we okay with this?
>
> Also if the device were to support at least 1 extra channel that can do
> (offloaded) off-channel we wouldn't have to do steps: 2, 3, 11.


Ok, so ... I think we're getting to the bottom of our misunderstanding &
confusion. I've completely ignored scan/off-channel in this because I
don't have to handle it in software, our device handles it itself. It
seems from this that you do need it handled in software.

The question is -- do we want mac80211 to handle it? I'll get back to
that later after I reply to the other points, see below.


> > Now that I think about it, the implicit way I previously thought was
> > quite a separate mechanism is really very similar: you'd still have to
> > have the assign_vif_channel_context function, maybe as part of
> > bss_info_changed, but not have explicit add/remove. I think the
> > add/remove is useful so let's not consider implicit methods any more :)
>
> I'm unsure about the add/remove. See above. We can't just add a channel
> context without stating the purpose of that channel context. A driver
> might need to know of the purpose beforehand.

Yes, I was only thinking of "for vif" contexts.


> > Wrt. off-channel, I'm not sure I've understood you correctly. The
> > current off-channel is a very short-lived thing that doesn't really use
> > a "complete" channel context. In fact, in our implementation it is
> > completely separate since it's temporary (**).
>
> Maybe you're right, but how can we then express off-channel in a neat
> clean way? If we use channel contexts we get software AND hardware
> off-channel (as long as the device supports it) at the same time.

I'm not sure. The detach logic etc. you proposed above will really not
be implementable in our device's context handling.


> If it's a concern that userspace may want to start a new interface on a
> different channel while off-channel is taking up "a precious channel
> context", we may either do -EBUSY or just sleep a few seconds/defer the
> work if the off-channel is really short-lived.

Yeah, *handwaving*, I wouldn't be concerned about that either :-)


> > (*) after we refactor cfg80211_can_change_interface() and export
> > cfg80211_get_current_combination() that returns the pointer to the
> > combination that's currently in use, or something like that. We'd have
> > to figure out if another combination is available that has more channels
> > though, if needed.
>
> Is there a lot of work to be done?

Nah, the biggest problem is probably figuring out what we need/want and
how we use it.


> > (**) However, we're thinking about implementing a P2P-device context
> > that would allow offloading things like periodic listen
> > (discoverability) or find (search + listen). That would probably use a
> > channel context then.
>
> But that wouldn't need a permanent channel context, would it? It'll
> probably be cheap to attach/detach channel contexts.

This would actually want a permanent context since you'd want the device
to schedule the periodic listen on that context.

We haven't really talked about this, but obviously there's going to have
to be some scheduler that tells the device how to switch between
multiple channel contexts, probably only by some sort of "quota".


> >> Should we add capability
> >> flags for each ieee80211_channel_stream? Or we could add new driver
> >> callback, eg. "set_channel_try" and allow the driver to decide
> >> whether it allows a certain hardware state or not.
> >
> > This is not a fully feasible "stream" anyway. Think of it more like a
> > single scan dwell -- you wouldn't express that as a separate "stream"
> > I think?
>
> Why not? Should we care what vif we're working with, or hw for that
> matter? Off-channel, as well as other operation means we want to do
> something on a certain channel. It shouldn't matter how long we use that
> channel (except if we're borrowing time from another operational mode).

I'm not sure how your device works, but it seems that the device would
require binding vifs to a channel context so that it knows which is
active where? When to synchronise with beacons, for example?


> Another question would be if we could do software multi-channel
> operation. Do you think it's even possible and worth exploring?

I don't think it's feasible to implement in mac80211 because of timing
constraints. With devices like ath9k it can probably be done by relying
on hardware timers, interrupts and queue blocking/opening, but because
mac80211 has to deal with USB and SDIO chips too which can be really
slow it can't really do this. So no, I wouldn't think it's worth
exploring at least not directly in mac80211 -- maybe as a separate
helper library or something.


Now let's go back to the question of off-channel/scan -- this seems to
be the fundamental difference in how we look at things. First, the same
thing applies here. SW scanning in mac80211 right now is very
inefficient when you try to do background scanning, since it doesn't
synchronise with beacons etc. With multiple virtual interfaces, this
will only get worse. With multiple channels, probably more so.

Then also I'm in a bit of a luxurious position -- our device implements
scanning and off-channel all by itself. For this reason, but also
because of the timing, I've always wanted to ignore off-channel and
scanning in the multi-channel code as much as possible.

Your proposal to add temporary channel contexts might be feasible, but I
wonder how complex it would become and how much of it can really be
shared? I expect that ath9k will implement multi-channel completely in
software, and any scheduler there would probably also roll in
scanning/off-channel. (Not that I know, I don't work for QCA :) )

Also, I'm a bit afraid that doing so would make the channel context API
even more complex, and it's not really easy anyway.

I think I need to know more about how you would need to use those
temporary channel contexts, but I have a feeling that we might be better
off by splitting it into a separate helper library. I'm thinking this
would essentially sit between mac80211 and the driver, and offer
"translation services", i.e. it would have a "remain_on_channel"
function that the driver could call for mac80211's remain_on_channel
callback, and then this separate code could do something like your
temporary channel context detach/attach etc.

Would that seem okay to you? I'd rather leave this complexity out of
mac80211, not just for complexity reasons but also to make the API have
easier semantics.

This helper library could also make more assumptions and have timing
inputs, so if asked to scan (the scan state machine would move into it)
it could drive the scan state machine off timing inputs from the
hardware rather than today's essentially random behaviour (wrt. beacons
etc.)

johannes


2012-03-23 15:01:37

by Johannes Berg

[permalink] [raw]
Subject: Re: [RFC 00/12] multi-channel support

On Fri, 2012-03-23 at 15:03 +0100, Michał Kazior wrote:

> What if we create a new structure that would be a channel
> stream/reservation? Driver could then report multiple such objects.
> Each such would have a tx queue and it's own locking. Let's assume
> something like:
>
> struct ieee80211_channel_stream {
> struct list_head vif; // list of vifs we're attached to
> struct ieee80211_channel *ch;
> struct sk_buff_head pending;
> int stop_reasons[IEEE80211_MAX_QUEUES];
> // .. other stuff
> u8 *drv_priv[0] __attribute__((__aligned__((void *))));
> };

I'm not sure I see the need to have the driver register these, but it
seems useful to maintain this abstraction somehow in mac80211.

Due to the interface combinations advertising, we already know (*) what
is supported by the driver, so there's no need for the driver to
register this again.

Like I said earlier, I see basically two ways of handling this.

The explicit way would essentially consist of something like this:

struct ieee80211_channel_context {
struct ieee80211_channel *chan;
enum nl80211_channel_type type;
};

with API to add/remove such contexts from/to the device:

struct ieee80211_ops {
...
void (*add_channel_context)(hw, ctx),
void (*remove_channel_context)(hw, ctx),
void (*change_channel_type)(hw, ctx),
void (*assign_vif_channel_context)(hw, vif, ctx),
};

The assign function would have rather strange semantics though -- it can
be called only when the interface is up (added to the driver), but it
can't be called while the interface is associated, active as an AP, etc.
EXCEPT ... maybe it can in CSA scenarios. But I said that before: CSA is
a bit of a mess to handle.

Now that I think about it, the implicit way I previously thought was
quite a separate mechanism is really very similar: you'd still have to
have the assign_vif_channel_context function, maybe as part of
bss_info_changed, but not have explicit add/remove. I think the
add/remove is useful so let's not consider implicit methods any more :)

The internal handling in mac80211 might have a linked list of the
channel contexts, or just a static array of a handful of them, doesn't
really matter. It would, however, make sure that there aren't more than
the current interface combination advertised as supported (see (*)
again.)

Of course mac80211 would also assign the interfaces to channel contexts
as needed -- if a given interface is already on channel 1, the next one
wouldn't need a new channel context if it also uses channel 1 and the
channel types are compatible. But that reminds me -- when we do that we
may have to update the channel context to change its channel type, so
add an operation for that.


Wrt. off-channel, I'm not sure I've understood you correctly. The
current off-channel is a very short-lived thing that doesn't really use
a "complete" channel context. In fact, in our implementation it is
completely separate since it's temporary (**).

As far as scanning is concerned, I'm not really sure. I haven't spent
much time thinking about it since we have hardware offload for it (and
also off-channel) -- I'm not sure what a good solution would be.

johannes


(*) after we refactor cfg80211_can_change_interface() and export
cfg80211_get_current_combination() that returns the pointer to the
combination that's currently in use, or something like that. We'd have
to figure out if another combination is available that has more channels
though, if needed.

(**) However, we're thinking about implementing a P2P-device context
that would allow offloading things like periodic listen
(discoverability) or find (search + listen). That would probably use a
channel context then.


2012-03-28 06:12:57

by Michal Kazior

[permalink] [raw]
Subject: Re: [RFC 00/12] multi-channel support

Johannes Berg wrote:
>>> Would that seem okay to you? I'd rather leave this complexity out of
>>> mac80211, not just for complexity reasons but also to make the API have
>>> easier semantics.
>>
>> I get the general idea but I can't imagine any detailed examples. I'm
>> yet to fully understand mac80211 and other drivers/devices.
>
> Ok. Basically I'm punting software-off-channel/-scan in multi-channel
> context -- is that something you could work with for your device? Does
> your device have hw-scan/hw-off-channel?

Yes, it can do hw-scan and hw-off-channel.


--
Pozdrawiam / Best regards, Michal Kazior.

2012-03-27 11:59:31

by Johannes Berg

[permalink] [raw]
Subject: Re: [RFC 00/12] multi-channel support

On Tue, 2012-03-27 at 13:42 +0200, Michal Kazior wrote:

> >> But how do we then check if we need to stop given queue or not? Let's
> >> say a driver stops q=2 which corresponds to BE on vif0 and vif1. But
> >> then comes mac80211 aggregation and stops BE on vif0. I can only think
> >> of a single solution to that: "u8 lock_reasons[256];" in ieee80211_local
> >> but it seems like an overkill or is it not?
> >
> > Essentially that's what I was considering, we'd maintain all the queue
> > state per HW queue ID. I said it would be a u8 for the HW queue ID, but
> > I don't see anyone using more than say 32 or so, so we wouldn't really
> > need 256. But we could go up to 256 if needed (though at that point we
> > should probably do dynamic allocation instead.)
>
> We could maybe allocate it according to hw.queues? I think it should be
> okay unless a driver has some kind of non-static queues. Are you aware
> of such a device/driver?

Yeah, we could do that. I'm not sure I want the complexity though, if
the limit is going to be something like 16 (I said 32 before but now
think for our current use 16 would be sufficient) I don't see much value
in dynamically allocating over just reserving space for 16.

johannes


2012-03-23 15:06:43

by Johannes Berg

[permalink] [raw]
Subject: Re: [RFC 00/12] multi-channel support

On Fri, 2012-03-23 at 16:01 +0100, Johannes Berg wrote:

> struct ieee80211_ops {
> ...
> void (*add_channel_context)(hw, ctx),
> void (*remove_channel_context)(hw, ctx),
> void (*change_channel_type)(hw, ctx),
> void (*assign_vif_channel_context)(hw, vif, ctx),
> };

And then we come full circle back to the queue mappings of course --
when the channel context is assigned, the driver is able to change the
queue details in the vif to map to whatever it needs, if anything
different. Like I said in the other email, some devices may use queues
for each interface regardless of channel (e.g. wl12xx, from discussions
with them) while others (ath9k at least, I suspect) would use queues per
channel context. For iwlwifi, I'm not sure yet.

johannes


2012-03-26 12:45:21

by Johannes Berg

[permalink] [raw]
Subject: Re: [RFC 00/12] multi-channel support

Hi,

> > I'm not sure we can easily use such a "stream" for off-channel
> > operation. In particular, that "stream" might not support multiple
> > queues.
>
> Does it need to? It could as well just map all queues to a single hw
> queue with this design. I'm not sure if I get your point here.
>
>
> Is it okay for us to not care if the driver supports real queues for
> ACs? Should we care if a driver maps all ACs from a channel context (or
> vif for that matter) onto a single hw queue? I'm wondering whether that
> would simplify code paths?

Well, ok, that would be possible, but I'm not sure I see the point in
using a stream for off-channel. Today, we completely offload off-channel
to the device, and I don't see how that could change? Having temporary
context allocations all the time seems like a lot of thrash?

> >> struct ieee80211_queue_types { // or tx_queue_types?
> >> IEEE80211_QUEUE_VO,
> >> IEEE80211_QUEUE_VI,
> >> IEEE80211_QUEUE_BE,
> >> IEEE80211_QUEUE_BK,
> >> IEEE80211_QUEUE_CAB,
> >>
> >> IEEE80211_QUEUE_LAST
> >> };
> > [...]
> >> This looks kind of ugly, but oh well. This way the driver will be able
> >> to setup mapping between mac80211 and internal device queues. It should
> >> also be possible to map one hardware queue to multiple
> >> ieee80211_channel_stream structures (queues is a list of pointers) so
> >> multiple vifs on the same channel is okay.
> >
> > I don't think this makes a lot of sense, tbh. This isn't a separate AC,
> > I'd rather keep them separated.
>
> You mean the CAB queue separated from ACs?

Yeah. I don't see CAB as a separate AC. It's just a separate queue.


> > I'm looking at the code right now, and I think we should treat ACs,
> > queues and channels separately. If we combine queue sets& channels into
> > one, I think we'll find drivers that don't work correctly with this.
> > Some devices for instance might have a set of queues for each interface,
> > even if they're on the same channel. I don't want to lose flexibility to
> > handle that.
>
> You mean a driver that can handle two vifs simultaneously but on the
> same channel and have two separate queue sets for that?

Yes, that's a possibility we have to consider, I think our device might
end up behaving like that.

> Then we need to move queues to ieee80211_vif and create callbacks to
> bind channels contexts with a vif (just you mentioned that in other email).

Right.


> > struct ieee80211_hw {
> > ...
> > u8 offchannel_queue;
> > };
> >
> > struct ieee80211_vif {
> > ...
> > u8 hw_queue[IEEE80211_NUM_ACS];
> > u8 after_dtim_queue;
> > };
> >
> > For any of these (offchannel_queue, after_dtim_queue, hw_queue[i]), the
> > queue number Q must be 0<= Q< hw.queues. This allows us to keep all
> > per-hw-queue state "flattened" in ieee80211_local's pending,
> > queue_stop_reasons, etc. instead of separating all of it out into
> > separate structures.
>
> So you want "offchannel_queue", "hw_queue[]" and "after_dtim_queue" to
> point to driver/device hw queue id?

The driver is free to point it to anything it want, but typically it
would point it to the real HW queue ID -- this would be the same ID it
has to give to ieee80211_stop_queue()/wake_queue() as well.


> > When a queue is stopped, we do:
> >
> > void queue_stop(int q)
> > {
> > for_each_interface(vif) {
> > if (vif->hw_queue[AC] == q) // for all 4 ACs
> > stop netdev queue (vif->dev, AC)
> > if (vif->after_dtim_queue == q)
> > stop_all_queues(vif->dev)
> > }
> > }
> >
> > Or so ... after_dtim_queue is kinda a special case.
>
> Oh, I see. So stopping CAB would be like "stop all the things".

CAB is really special -- not sure what the best thing here is. But
technically frames from any AC can be multicast so I suppose this is the
best way to handle it after all. I wouldn't really worry too much about
CAB though :)


> But how do we then check if we need to stop given queue or not? Let's
> say a driver stops q=2 which corresponds to BE on vif0 and vif1. But
> then comes mac80211 aggregation and stops BE on vif0. I can only think
> of a single solution to that: "u8 lock_reasons[256];" in ieee80211_local
> but it seems like an overkill or is it not?

Essentially that's what I was considering, we'd maintain all the queue
state per HW queue ID. I said it would be a u8 for the HW queue ID, but
I don't see anyone using more than say 32 or so, so we wouldn't really
need 256. But we could go up to 256 if needed (though at that point we
should probably do dynamic allocation instead.)


johannes