Hi,
I'm currently running a number of sites using the RT61 WLAN chipset.
>From time to time we have fatal roaming problems.
I have collected a trace that show normal and problem situation.
Any help explaining the cause of the problem is appreciated.
Below is a summary of the trace. Attached it the complete trace.
/Lars
CASE 1: The above trace is a typical trace from a roaming atempt.
==========================================================
361830.763657 We lost conatct with current AP 00:0e:d7:ac:84:20
361831.252992 We have selected the new AP 00:0f:24:d1:5e:e0
361831.256287 Why do we try to authenticate with previous AP
00:0e:d7:ac:84:20 ? <-----
361831.261071 Finaly we selected the correct AP 00:0f:24:d1:5e:e0
361831.295573 We are up and running
CASE 2: The above trace is a typical trace from a roaming atempt.
==========================================================
362123.346748 We lost conatct with current AP 00:0f:24:d1:5e:e0
362123.842242 We have selected the new AP 00:0e:d7:ac:84:20
362123.861586 We try connect AP 00:0e:d7:ac:84:20
362123.874442 We are up and running
CASE 3: The above trace is a typical trace from a roaming atempt.
===================================================================
362441.946825 We lost conatct with current AP 00:0e:d7:ac:84:20
362442.449435 We have selected the new AP 00:0f:24:a3:b8:60
362442.466944 Why do we try to authenticate with previous AP
00:0e:d7:ac:84:20 ? <-----
362442.470147 Finaly we selected the correct AP 00:0f:24:a3:b8:60
361831.295573 We are up and running
362443.066379 Timeout.
362444.886776 New try with selected AP 00:0f:24:a3:b8:60
362444.904429 Associated
362444.904883 What initiate this 'Initial auth_alg=0' with the associated
AP ? <-----
362444.908677 What initiate this 'Initial auth_alg=0' with AP
00:0f:24:d1:5e:e0 ? <-----
362445.505964 No one reacts on the 'timed out' event <-----
On Wed, 2008-04-23 at 16:59 +0200, Johannes Berg wrote:
> > > > + // make sure no association start before we got a new BSSID
> > > > + ifsta->flags &= ~IEEE80211_STA_BSSID_SET;
> > >
> > > I don't think that patch makes sense, after all, userspace could request
> > > to disassociate and afterwards re-request to associate by setting the
> > > SSID and not setting the BSSID again, which would lose the fixed BSSID
> > > without userspace interaction.
> > >
> > > However, I'm not sure how to fix this.
> >
> > Enhance roaming support in wpa_supplicant I'd expect. If you set the
> > BSSID explicitly, which wpa_supplicant does, then the driver should not
> > roam to any other BSSID until userspace sends SIOCSIWAP
> > 00:00:00:00:00:00 or sets whatever the 'disabled' flag is. Such is
> > WEXT.
> >
> > I'm not actually sure why wpa_supplicant sets the BSSID explicitly, but
> > I also haven't looked into it much.
>
> Lars mentioned that it sets the SSID first and then the BSSID, maybe the
> trivial fix would be to make it do that the other way around?
Maybe, but why should the driver care? It should be re-attempting
association when either of SSID or BSSID gets set, interrupting any
current ongoing association to a different AP.
Dan
> > > + // make sure no association start before we got a new BSSID
> > > + ifsta->flags &= ~IEEE80211_STA_BSSID_SET;
> >
> > I don't think that patch makes sense, after all, userspace could request
> > to disassociate and afterwards re-request to associate by setting the
> > SSID and not setting the BSSID again, which would lose the fixed BSSID
> > without userspace interaction.
> >
> > However, I'm not sure how to fix this.
>
> Enhance roaming support in wpa_supplicant I'd expect. If you set the
> BSSID explicitly, which wpa_supplicant does, then the driver should not
> roam to any other BSSID until userspace sends SIOCSIWAP
> 00:00:00:00:00:00 or sets whatever the 'disabled' flag is. Such is
> WEXT.
>
> I'm not actually sure why wpa_supplicant sets the BSSID explicitly, but
> I also haven't looked into it much.
Lars mentioned that it sets the SSID first and then the BSSID, maybe the
trivial fix would be to make it do that the other way around?
johannes
> > I'm not disputing that as a reasonable approach. But FWIW for two
> > years we've been saying it is OK for drivers to use setting SSID as
> > the only trigger for association. Is that not your understanding?
>
Sorry, I din not know that there has been a discussion about the feature of
letting 'SSID set' trigger an association.
It's OK, but then the wpa_supplicant needs to 'cancel' current settings in
the driver before it start prepare for the new AP.
As I have understood it, that is done by issuing a BSSID of
00:00:00:00:00:00. I will look into a patch for that.
An other aspect of the selected design. If the driver (STA) is associated
with an BSSID and the wpa_supplicant would like to change BSSID, who is
responsible for a proper de- association/authentication from the first BSSID
?
My findings says that the network does not allow an STA to
associate/authenticate with one BSSID while it still have a valid aid with
an other BSSID. At least my Cisco AP does not allow that.
/Lars
> > The below patch makes sure that we drop the BSSID when we
> disassociate.
>
> > --- a/ieee80211_sta.c Wed Apr 23 10:14:30 2008
> > +++ b/ieee80211_sta.c Wed Apr 23 08:41:23 2008
> > @@ -479,6 +479,9 @@ static void ieee80211_set_associated(str
> > netif_carrier_off(dev);
> > ieee80211_reset_erp_info(dev);
> > memset(wrqu.ap_addr.sa_data, 0, ETH_ALEN);
> > +
> > + // make sure no association start before we got
> a new BSSID
> > + ifsta->flags &= ~IEEE80211_STA_BSSID_SET;
>
> I don't think that patch makes sense, after all, userspace
> could request to disassociate and afterwards re-request to
> associate by setting the SSID and not setting the BSSID
> again, which would lose the fixed BSSID without userspace interaction.
>
> However, I'm not sure how to fix this.
>
Ok, I had that in mind but could not see how to solve that.
The root problem is that the drive start associate when any of the
ieee80211_ioctl_siwgenie()
ieee80211_ioctl_siwessid() and ieee80211_ioctl_siwap() is called.
If the wpa_supplicant need call more than one of them the association start
to early.
I can see two ways to solve this:
1) Add an argument/flag to the ioctls above indicating if association should
start.
2) Add a new ioctrl that initiate the association.
My patch above relies on that the wpa_supplicant always end the AP setup
with a ieee80211_ioctl_siwap().
/Lars
> Maybe, but why should the driver care? It should be re-attempting
> association when either of SSID or BSSID gets set, interrupting any
> current ongoing association to a different AP.
Apparently the "interrupting any current ongoing association" bit isn't
right.
johannes
> The below patch makes sure that we drop the BSSID when we disassociate.
> --- a/ieee80211_sta.c Wed Apr 23 10:14:30 2008
> +++ b/ieee80211_sta.c Wed Apr 23 08:41:23 2008
> @@ -479,6 +479,9 @@ static void ieee80211_set_associated(str
> netif_carrier_off(dev);
> ieee80211_reset_erp_info(dev);
> memset(wrqu.ap_addr.sa_data, 0, ETH_ALEN);
> +
> + // make sure no association start before we got a new BSSID
> + ifsta->flags &= ~IEEE80211_STA_BSSID_SET;
I don't think that patch makes sense, after all, userspace could request
to disassociate and afterwards re-request to associate by setting the
SSID and not setting the BSSID again, which would lose the fixed BSSID
without userspace interaction.
However, I'm not sure how to fix this.
johannes
On Mon, 2008-04-28 at 20:28 +0200, Lars Ericsson wrote:
> > > - Driver authenticating with new AP while having a valid
> > > association with and other AP.
> >
> > Yes, that is the problem, we should disassociate first. If
> > you can come up with a patch to send a disassoc packet in
> > that case (and test it,
> > preferably) that would be much welcome.
>
> OK. I understood that there has been a discussion about the way the driver
> should handle this situation and the current behaviour is according to that
> discussion.
Yeah but that was mostly just sidetrack.
> Is there any information document available describing the
> selected design path or is it only the code that document the design ?
> I think need that information if I should be able to create the patch
> requested above.
Unfortunately, there is nothing about mac80211's mlme other than the
code.
> On the other hand, that patch will only keep track of 'dual association'
> problem.
Well, no, that would mean we disassociate first and then only have a
single association.
> With the current wpa_supplicant, we will still have the problem
> with the driver start an association sequence as soon as 'trig' ioctl
> arrives. I think the wpa_supplicant need to modified to better cooperate
> with the driver.
That would be doable as well (though I would argue it should be done in
addition), we could use Holger's approach and delay the actual
association sequence start until a commit wext ioctl is received or a
timer expires. Then wpa_supplicant can call the commit explicitly after
doing everything.
johannes
Hi,
I think I have found the problem.
When the wpa_supplicant configure the device it issues a number of ioctls.
Some of them, ieee80211_ioctl_siwgenie()
ieee80211_ioctl_siwessid() and ieee80211_ioctl_siwap() initiates a
ieee80211_sta_req_auth().
If there is a valid BSSID, ieee80211_sta_req_auth() will post a authenticate
work.
First roaming attempt, no problem since we do not have any valid BSSID.
All following roaming will have the old BSSID when the wpa_supplicant start
preparing for next AP.
So any of the ieee80211_ioctl_siwgenie() or ieee80211_ioctl_siwessid() will
force the mac80211 to start associate with the old BSSID. First when the
ieee80211_ioctl_siwap() arrives, mac80211 knows which AP to use.
If, by timing, the erroneous first association success before the real AP
arrives, the real one may fail, with an infinite 'timed out' as a result.
The below patch makes sure that we drop the BSSID when we disassociate.
/Lars
============================================================================
====
--- a/ieee80211_sta.c Wed Apr 23 10:14:30 2008
+++ b/ieee80211_sta.c Wed Apr 23 08:41:23 2008
@@ -479,6 +479,9 @@ static void ieee80211_set_associated(str
netif_carrier_off(dev);
ieee80211_reset_erp_info(dev);
memset(wrqu.ap_addr.sa_data, 0, ETH_ALEN);
+
+ // make sure no association start before we got a new BSSID
+ ifsta->flags &= ~IEEE80211_STA_BSSID_SET;
}
wrqu.ap_addr.sa_family = ARPHRD_ETHER;
wireless_send_event(dev, SIOCGIWAP, &wrqu, NULL);
============================================================================
====
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Lars Ericsson
Sent: den 22 april 2008 17:09
To: [email protected]; [email protected]
Subject: Roaming problems
Hi,
I'm currently running a number of sites using the RT61 WLAN chipset.
>From time to time we have fatal roaming problems.
I have collected a trace that show normal and problem situation.
Any help explaining the cause of the problem is appreciated.
Below is a summary of the trace. Attached it the complete trace.
/Lars
CASE 1: The above trace is a typical trace from a roaming atempt.
==========================================================
361830.763657 We lost conatct with current AP 00:0e:d7:ac:84:20
361831.252992 We have selected the new AP 00:0f:24:d1:5e:e0
361831.256287 Why do we try to authenticate with previous AP
00:0e:d7:ac:84:20 ? <-----
361831.261071 Finaly we selected the correct AP 00:0f:24:d1:5e:e0
361831.295573 We are up and running
CASE 2: The above trace is a typical trace from a roaming atempt.
==========================================================
362123.346748 We lost conatct with current AP 00:0f:24:d1:5e:e0
362123.842242 We have selected the new AP 00:0e:d7:ac:84:20
362123.861586 We try connect AP 00:0e:d7:ac:84:20
362123.874442 We are up and running
CASE 3: The above trace is a typical trace from a roaming atempt.
===================================================================
362441.946825 We lost conatct with current AP 00:0e:d7:ac:84:20
362442.449435 We have selected the new AP 00:0f:24:a3:b8:60
362442.466944 Why do we try to authenticate with previous AP
00:0e:d7:ac:84:20 ? <-----
362442.470147 Finaly we selected the correct AP 00:0f:24:a3:b8:60
361831.295573 We are up and running
362443.066379 Timeout.
362444.886776 New try with selected AP 00:0f:24:a3:b8:60
362444.904429 Associated
362444.904883 What initiate this 'Initial auth_alg=0' with the associated
AP ? <-----
362444.908677 What initiate this 'Initial auth_alg=0' with AP
00:0f:24:d1:5e:e0 ? <-----
362445.505964 No one reacts on the 'timed out' event <-----
> I think the best way would be check the trace I attached in the first post.
I think the best way would be if you'd create an additional monitor
interface (with iw:
http://linuxwireless.sipsolutions.net/en/users/Documentation/iw) and
then make a packet capture on that interface while getting the problem.
johannes
> That would be doable as well (though I would argue it should
> be done in addition), we could use Holger's approach and
> delay the actual association sequence start until a commit
> wext ioctl is received or a timer expires. Then
> wpa_supplicant can call the commit explicitly after doing everything.
Yes I like this. A commit ioctl issued from the wpa_supplicant when all
configuration is done is the preferred solution.
The commit ioctl that will speed up the roaming and the timer that will trig
the association if no commit ioctl is received which gives backwards
compatibility.
But, we then need to change the current implementation where 3 ioctls trigs
an immediate association start.
/Lars
> No, wpa_supplicant shouldn't have to do that, mac80211 should
> do that automatically once a new BSSID comes in. The problem
> seems to be between
> mac80211 and your AP in that mac80211 associates to another
> AP and then tries the target one but the target one doesn't
> like that so we should at least try to disassociate from the
> previous one first.
>
>
> mac80211 should be responsible.
>
> I tend to think that is rather questionable behaviour, how
> can the second AP know that you're still in range to the
> first and can even disassociate from that one?
>
I have edited a trace/comment PDF file. I hope that will explain what I
seen.
It shows my major concerns:
- Driver initially authenticate with wrong AP
- Driver authenticating with new AP while having a valid association with
and other AP.
/Lars
On Mon, 2008-04-28 at 20:34 +0200, Johannes Berg wrote:
> On Mon, 2008-04-28 at 20:28 +0200, Lars Ericsson wrote:
> > > > - Driver authenticating with new AP while having a valid
> > > > association with and other AP.
> > >
> > > Yes, that is the problem, we should disassociate first. If
> > > you can come up with a patch to send a disassoc packet in
> > > that case (and test it,
> > > preferably) that would be much welcome.
> >
> > OK. I understood that there has been a discussion about the way the driver
> > should handle this situation and the current behaviour is according to that
> > discussion.
>
> Yeah but that was mostly just sidetrack.
>
> > Is there any information document available describing the
> > selected design path or is it only the code that document the design ?
> > I think need that information if I should be able to create the patch
> > requested above.
>
> Unfortunately, there is nothing about mac80211's mlme other than the
> code.
>
> > On the other hand, that patch will only keep track of 'dual association'
> > problem.
>
> Well, no, that would mean we disassociate first and then only have a
> single association.
>
> > With the current wpa_supplicant, we will still have the problem
> > with the driver start an association sequence as soon as 'trig' ioctl
> > arrives. I think the wpa_supplicant need to modified to better cooperate
> > with the driver.
>
> That would be doable as well (though I would argue it should be done in
> addition), we could use Holger's approach and delay the actual
> association sequence start until a commit wext ioctl is received or a
> timer expires. Then wpa_supplicant can call the commit explicitly after
> doing everything.
This is the best approach really; it's the one I took in the libertas
driver and turned out to work quite well. It completely avoids issues
with iwconfig argument ordering and the timer is long enough that it'll
pick up almost everything you can throw at it programmatically as long
as you don't do something stupid like sleep(2)...
Dan
> > Currently this is what happens most of the time, when all
> ioclts had
> > been issued from the driver, it drops current
> > authentication/association atempts and start with the new
> one. Most of the times that works fine.
> >
> > But sometimes, when for any reason the 3 ioctls get delayed, the
> > driver may success to associate with the old BSSID (since that is
> > still valid in the driver). Then when all 3 ioctls has been
> received,
> > including the new BSSID, the driver re-starts the
> > authentication/association sequence with the new BSSID.
> >
> > The problem is that the STA already have a valid
> association with the
> > first AP. In this case the new BSSID silently ignores any
> > authentication/association attempts. My believe is that it
> knows that
> > the STA already have a valid association.
>
> Uh huh. Aren't we disassociating when we change to a new bssid?
>
I think the best way would be check the trace I attached in the first post.
/LaE
> - Driver authenticating with new AP while having a valid association with
> and other AP.
Yes, that is the problem, we should disassociate first. If you can come
up with a patch to send a disassoc packet in that case (and test it,
preferably) that would be much welcome.
johannes
On Wed, 2008-04-23 at 18:27 +0200, Lars Ericsson wrote:
> > > The below patch makes sure that we drop the BSSID when we
> > disassociate.
> >
> > > --- a/ieee80211_sta.c Wed Apr 23 10:14:30 2008
> > > +++ b/ieee80211_sta.c Wed Apr 23 08:41:23 2008
> > > @@ -479,6 +479,9 @@ static void ieee80211_set_associated(str
> > > netif_carrier_off(dev);
> > > ieee80211_reset_erp_info(dev);
> > > memset(wrqu.ap_addr.sa_data, 0, ETH_ALEN);
> > > +
> > > + // make sure no association start before we got
> > a new BSSID
> > > + ifsta->flags &= ~IEEE80211_STA_BSSID_SET;
> >
> > I don't think that patch makes sense, after all, userspace
> > could request to disassociate and afterwards re-request to
> > associate by setting the SSID and not setting the BSSID
> > again, which would lose the fixed BSSID without userspace interaction.
> >
> > However, I'm not sure how to fix this.
> >
>
> Ok, I had that in mind but could not see how to solve that.
>
> The root problem is that the drive start associate when any of the
> ieee80211_ioctl_siwgenie()
> ieee80211_ioctl_siwessid() and ieee80211_ioctl_siwap() is called.
> If the wpa_supplicant need call more than one of them the association start
> to early.
When either SIWESSID or SIWAP is called, the driver should restart
association using the latest settings sent by the supplicant. So by the
end of the GENIE -> SSID -> BSSID call chain, the driver should have
everything it needs to associate, and should restart association with
those parameters.
Dan
> I'm not disputing that as a reasonable approach. But FWIW for
> two years we've been saying it is OK for drivers to use
> setting SSID as the only trigger for association. Is that not
> your understanding?
I have a patch for better-reassociation-for-libertas, where both
SIWESSID or SIWAP trigger an re-association --- but with a delay
of some hundred milliseconds. Works great with with iwconfig and
wpa_supplicant.
(However, I won't share this patch yet because it is completely
AdHoc-agnostic).
On Wed, Apr 23, 2008 at 12:31:14PM -0400, Dan Williams wrote:
> On Wed, 2008-04-23 at 18:27 +0200, Lars Ericsson wrote:
> > The root problem is that the drive start associate when any of the
> > ieee80211_ioctl_siwgenie()
> > ieee80211_ioctl_siwessid() and ieee80211_ioctl_siwap() is called.
> > If the wpa_supplicant need call more than one of them the association start
> > to early.
>
> When either SIWESSID or SIWAP is called, the driver should restart
> association using the latest settings sent by the supplicant. So by the
> end of the GENIE -> SSID -> BSSID call chain, the driver should have
> everything it needs to associate, and should restart association with
> those parameters.
I'm not disputing that as a reasonable approach. But FWIW for two
years we've been saying it is OK for drivers to use setting SSID as
the only trigger for association. Is that not your understanding?
Clearly we need to get rolling on doing associations through nl80211...
John
--
John W. Linville
[email protected]
On Wed, 2008-04-23 at 18:44 +0200, Lars Ericsson wrote:
> Currently this is what happens most of the time, when all ioclts had been
> issued from the driver, it drops current authentication/association atempts
> and start with the new one. Most of the times that works fine.
>
> But sometimes, when for any reason the 3 ioctls get delayed, the driver may
> success to associate with the old BSSID (since that is still valid in the
> driver). Then when all 3 ioctls has been received, including the new BSSID,
> the driver re-starts the authentication/association sequence with the new
> BSSID.
>
> The problem is that the STA already have a valid association with the first
> AP. In this case the new BSSID silently ignores any
> authentication/association attempts. My believe is that it knows that the
> STA already have a valid association.
Uh huh. Aren't we disassociating when we change to a new bssid?
johannes
> > Lars mentioned that it sets the SSID first and then the BSSID, maybe
> > the trivial fix would be to make it do that the other way around?
>
> Maybe, but why should the driver care? It should be
> re-attempting association when either of SSID or BSSID gets
> set, interrupting any current ongoing association to a different AP.
>
> Dan
>
Currently this is what happens most of the time, when all ioclts had been
issued from the driver, it drops current authentication/association atempts
and start with the new one. Most of the times that works fine.
But sometimes, when for any reason the 3 ioctls get delayed, the driver may
success to associate with the old BSSID (since that is still valid in the
driver). Then when all 3 ioctls has been received, including the new BSSID,
the driver re-starts the authentication/association sequence with the new
BSSID.
The problem is that the STA already have a valid association with the first
AP. In this case the new BSSID silently ignores any
authentication/association attempts. My believe is that it knows that the
STA already have a valid association.
Could the problem be that the wpa_supplicant not issue a BSSID
00:00:00:00:00:00 before start the configuration to the new BSSID ?
/Lars
> It's OK, but then the wpa_supplicant needs to 'cancel' current settings in
> the driver before it start prepare for the new AP.
> As I have understood it, that is done by issuing a BSSID of
> 00:00:00:00:00:00. I will look into a patch for that.
No, wpa_supplicant shouldn't have to do that, mac80211 should do that
automatically once a new BSSID comes in. The problem seems to be between
mac80211 and your AP in that mac80211 associates to another AP and then
tries the target one but the target one doesn't like that so we should
at least try to disassociate from the previous one first.
> An other aspect of the selected design. If the driver (STA) is associated
> with an BSSID and the wpa_supplicant would like to change BSSID, who is
> responsible for a proper de- association/authentication from the first BSSID
> ?
mac80211 should be responsible.
> My findings says that the network does not allow an STA to
> associate/authenticate with one BSSID while it still have a valid aid with
> an other BSSID. At least my Cisco AP does not allow that.
I tend to think that is rather questionable behaviour, how can the
second AP know that you're still in range to the first and can even
disassociate from that one?
johannes
On Wed, 2008-04-23 at 13:12 +0200, Johannes Berg wrote:
> > The below patch makes sure that we drop the BSSID when we disassociate.
>
> > --- a/ieee80211_sta.c Wed Apr 23 10:14:30 2008
> > +++ b/ieee80211_sta.c Wed Apr 23 08:41:23 2008
> > @@ -479,6 +479,9 @@ static void ieee80211_set_associated(str
> > netif_carrier_off(dev);
> > ieee80211_reset_erp_info(dev);
> > memset(wrqu.ap_addr.sa_data, 0, ETH_ALEN);
> > +
> > + // make sure no association start before we got a new BSSID
> > + ifsta->flags &= ~IEEE80211_STA_BSSID_SET;
>
> I don't think that patch makes sense, after all, userspace could request
> to disassociate and afterwards re-request to associate by setting the
> SSID and not setting the BSSID again, which would lose the fixed BSSID
> without userspace interaction.
>
> However, I'm not sure how to fix this.
Enhance roaming support in wpa_supplicant I'd expect. If you set the
BSSID explicitly, which wpa_supplicant does, then the driver should not
roam to any other BSSID until userspace sends SIOCSIWAP
00:00:00:00:00:00 or sets whatever the 'disabled' flag is. Such is
WEXT.
I'm not actually sure why wpa_supplicant sets the BSSID explicitly, but
I also haven't looked into it much.
Dan
> > - Driver authenticating with new AP while having a valid
> > association with and other AP.
>
> Yes, that is the problem, we should disassociate first. If
> you can come up with a patch to send a disassoc packet in
> that case (and test it,
> preferably) that would be much welcome.
OK. I understood that there has been a discussion about the way the driver
should handle this situation and the current behaviour is according to that
discussion. Is there any information document available describing the
selected design path or is it only the code that document the design ?
I think need that information if I should be able to create the patch
requested above.
On the other hand, that patch will only keep track of 'dual association'
problem. With the current wpa_supplicant, we will still have the problem
with the driver start an association sequence as soon as 'trig' ioctl
arrives. I think the wpa_supplicant need to modified to better cooperate
with the driver.
The patch that I suggested is now installed in the system where I had the
problem. The traces looks good and I have had no problems reported so far.
/Lars