2016-03-03 22:37:32

by Rafał Miłecki

[permalink] [raw]
Subject: Problem with brcmfmac removing extra interface

Hi,

brcmfmac in general is not capable of removing interfaces. If you take
a look at brcmf_cfg80211_del_iface implementation, you'll see it
returns -EOPNOTSUPP (except for p2p interfaces).

However there is problem in handling NL80211_CMD_STOP_AP (with the
brcmf_cfg80211_stop_ap callback). Current implementation calls
brcmf_fil_cmd_int_set(ifp, BRCMF_C_DOWN, 1)
if mbss support is enabled/used.

Above call results in firmware generating BRCMF_E_LINK event. This
event is handled with the following forward-traced functions chain:
1) brcmf_notify_connect_status
2) brcmf_notify_connect_status_ap
3) brcmf_remove_interface
4) brcmf_del_if
5) brcmf_net_detach
6) unregister_netdev

So the result of NL80211_CMD_STOP_AP is interface being removed. The
problem with this behavior is that interface can't be recreated after
that:
# iw phy phy1 interface add wlan1-1 type __ap
[ 3602.929199] brcmfmac: brcmf_ap_add_vif: timeout occurred
command failed: I/O error (-5)

I hit this problem when trying to restart hostapd using BCM43602 and 2 BSSes.

Could you analyze this problem and see if you see some way of solving
this problem, please?

--
Rafał


2016-03-22 06:36:39

by Rafał Miłecki

[permalink] [raw]
Subject: Re: Problem with brcmfmac removing extra interface

On 3 March 2016 at 23:37, Rafał Miłecki <[email protected]> wrote:
> brcmfmac in general is not capable of removing interfaces. If you take
> a look at brcmf_cfg80211_del_iface implementation, you'll see it
> returns -EOPNOTSUPP (except for p2p interfaces).
>
> However there is problem in handling NL80211_CMD_STOP_AP (with the
> brcmf_cfg80211_stop_ap callback). Current implementation calls
> brcmf_fil_cmd_int_set(ifp, BRCMF_C_DOWN, 1)
> if mbss support is enabled/used.
>
> Above call results in firmware generating BRCMF_E_LINK event. This
> event is handled with the following forward-traced functions chain:
> 1) brcmf_notify_connect_status
> 2) brcmf_notify_connect_status_ap
> 3) brcmf_remove_interface
> 4) brcmf_del_if
> 5) brcmf_net_detach
> 6) unregister_netdev
>
> So the result of NL80211_CMD_STOP_AP is interface being removed. The
> problem with this behavior is that interface can't be recreated after
> that:
> # iw phy phy1 interface add wlan1-1 type __ap
> [ 3602.929199] brcmfmac: brcmf_ap_add_vif: timeout occurred
> command failed: I/O error (-5)
>
> I hit this problem when trying to restart hostapd using BCM43602 and 2 BSSes.
>
> Could you analyze this problem and see if you see some way of solving
> this problem, please?

Hi, any ideas / help regarding this issue?

--
Rafał

2016-03-22 20:24:10

by Arend Van Spriel

[permalink] [raw]
Subject: Re: Problem with brcmfmac removing extra interface

On 22-3-2016 7:36, Rafał Miłecki wrote:
> On 3 March 2016 at 23:37, Rafał Miłecki <[email protected]> wrote:
>> brcmfmac in general is not capable of removing interfaces. If you take
>> a look at brcmf_cfg80211_del_iface implementation, you'll see it
>> returns -EOPNOTSUPP (except for p2p interfaces).
>>
>> However there is problem in handling NL80211_CMD_STOP_AP (with the
>> brcmf_cfg80211_stop_ap callback). Current implementation calls
>> brcmf_fil_cmd_int_set(ifp, BRCMF_C_DOWN, 1)
>> if mbss support is enabled/used.
>>
>> Above call results in firmware generating BRCMF_E_LINK event. This
>> event is handled with the following forward-traced functions chain:
>> 1) brcmf_notify_connect_status
>> 2) brcmf_notify_connect_status_ap
>> 3) brcmf_remove_interface
>> 4) brcmf_del_if
>> 5) brcmf_net_detach
>> 6) unregister_netdev
>>
>> So the result of NL80211_CMD_STOP_AP is interface being removed. The
>> problem with this behavior is that interface can't be recreated after
>> that:
>> # iw phy phy1 interface add wlan1-1 type __ap
>> [ 3602.929199] brcmfmac: brcmf_ap_add_vif: timeout occurred
>> command failed: I/O error (-5)
>>
>> I hit this problem when trying to restart hostapd using BCM43602 and 2 BSSes.
>>
>> Could you analyze this problem and see if you see some way of solving
>> this problem, please?
>
> Hi, any ideas / help regarding this issue?

Restarting hostapd obviously is a valid scenario. My guess is we would
need to avoid interface removal in brcmf_notify_connect_status_ap. But
first I would like to know in which state the firmware is upon
brcmf_ap_add_vif. Can you provide a full log with FWCON debug level enabled?

Regards,
Arend

2016-03-23 08:47:48

by Rafał Miłecki

[permalink] [raw]
Subject: Re: Problem with brcmfmac removing extra interface

On 22 March 2016 at 21:24, Arend van Spriel
<[email protected]> wrote:
> On 22-3-2016 7:36, Rafał Miłecki wrote:
>> On 3 March 2016 at 23:37, Rafał Miłecki <[email protected]> wrote:
>>> brcmfmac in general is not capable of removing interfaces. If you take
>>> a look at brcmf_cfg80211_del_iface implementation, you'll see it
>>> returns -EOPNOTSUPP (except for p2p interfaces).
>>>
>>> However there is problem in handling NL80211_CMD_STOP_AP (with the
>>> brcmf_cfg80211_stop_ap callback). Current implementation calls
>>> brcmf_fil_cmd_int_set(ifp, BRCMF_C_DOWN, 1)
>>> if mbss support is enabled/used.
>>>
>>> Above call results in firmware generating BRCMF_E_LINK event. This
>>> event is handled with the following forward-traced functions chain:
>>> 1) brcmf_notify_connect_status
>>> 2) brcmf_notify_connect_status_ap
>>> 3) brcmf_remove_interface
>>> 4) brcmf_del_if
>>> 5) brcmf_net_detach
>>> 6) unregister_netdev
>>>
>>> So the result of NL80211_CMD_STOP_AP is interface being removed. The
>>> problem with this behavior is that interface can't be recreated after
>>> that:
>>> # iw phy phy1 interface add wlan1-1 type __ap
>>> [ 3602.929199] brcmfmac: brcmf_ap_add_vif: timeout occurred
>>> command failed: I/O error (-5)
>>>
>>> I hit this problem when trying to restart hostapd using BCM43602 and 2 BSSes.
>>>
>>> Could you analyze this problem and see if you see some way of solving
>>> this problem, please?
>>
>> Hi, any ideas / help regarding this issue?
>
> Restarting hostapd obviously is a valid scenario. My guess is we would
> need to avoid interface removal in brcmf_notify_connect_status_ap. But
> first I would like to know in which state the firmware is upon
> brcmf_ap_add_vif. Can you provide a full log with FWCON debug level enabled?

Sure, hope it helps!

--
Rafał


Attachments:
brcmfmac-fwcon.txt (11.61 kB)

2016-03-23 19:14:11

by Rafał Miłecki

[permalink] [raw]
Subject: Re: Problem with brcmfmac removing extra interface

On 23 March 2016 at 11:44, Arend Van Spriel
<[email protected]> wrote:
> On 23-3-2016 9:47, Rafał Miłecki wrote:
>> On 22 March 2016 at 21:24, Arend van Spriel
>> <[email protected]> wrote:
>>> On 22-3-2016 7:36, Rafał Miłecki wrote:
>>>> Hi, any ideas / help regarding this issue?
>>>
>>> Restarting hostapd obviously is a valid scenario. My guess is we would
>>> need to avoid interface removal in brcmf_notify_connect_status_ap. But
>>> first I would like to know in which state the firmware is upon
>>> brcmf_ap_add_vif. Can you provide a full log with FWCON debug level enabled?
>>
>> Sure, hope it helps!
>
> Ah. Actually would like to see some driver logging as well so
> 'debug=0x101410'.

Sure, there you go.

--
Rafał


Attachments:
2016-03-23-brcmfmac-debug.txt (62.82 kB)

2016-03-23 10:45:07

by Arend Van Spriel

[permalink] [raw]
Subject: Re: Problem with brcmfmac removing extra interface

On 23-3-2016 9:47, Rafał Miłecki wrote:
> On 22 March 2016 at 21:24, Arend van Spriel
> <[email protected]> wrote:
>> On 22-3-2016 7:36, Rafał Miłecki wrote:
>>> On 3 March 2016 at 23:37, Rafał Miłecki <[email protected]> wrote:
>>>> brcmfmac in general is not capable of removing interfaces. If you take
>>>> a look at brcmf_cfg80211_del_iface implementation, you'll see it
>>>> returns -EOPNOTSUPP (except for p2p interfaces).
>>>>
>>>> However there is problem in handling NL80211_CMD_STOP_AP (with the
>>>> brcmf_cfg80211_stop_ap callback). Current implementation calls
>>>> brcmf_fil_cmd_int_set(ifp, BRCMF_C_DOWN, 1)
>>>> if mbss support is enabled/used.
>>>>
>>>> Above call results in firmware generating BRCMF_E_LINK event. This
>>>> event is handled with the following forward-traced functions chain:
>>>> 1) brcmf_notify_connect_status
>>>> 2) brcmf_notify_connect_status_ap
>>>> 3) brcmf_remove_interface
>>>> 4) brcmf_del_if
>>>> 5) brcmf_net_detach
>>>> 6) unregister_netdev
>>>>
>>>> So the result of NL80211_CMD_STOP_AP is interface being removed. The
>>>> problem with this behavior is that interface can't be recreated after
>>>> that:
>>>> # iw phy phy1 interface add wlan1-1 type __ap
>>>> [ 3602.929199] brcmfmac: brcmf_ap_add_vif: timeout occurred
>>>> command failed: I/O error (-5)
>>>>
>>>> I hit this problem when trying to restart hostapd using BCM43602 and 2 BSSes.
>>>>
>>>> Could you analyze this problem and see if you see some way of solving
>>>> this problem, please?
>>>
>>> Hi, any ideas / help regarding this issue?
>>
>> Restarting hostapd obviously is a valid scenario. My guess is we would
>> need to avoid interface removal in brcmf_notify_connect_status_ap. But
>> first I would like to know in which state the firmware is upon
>> brcmf_ap_add_vif. Can you provide a full log with FWCON debug level enabled?
>
> Sure, hope it helps!

Ah. Actually would like to see some driver logging as well so
'debug=0x101410'.

> 3) killall hostapd
> [ 234.424528] device wlan1-1 left promiscuous mode
> [ 234.429453] br-lan: port 3(wlan1-1) entered disabled state
> [ 234.842185] brcmfmac: CONSOLE: 026816.727 wl0: link down (wl0)
> [ 234.842224] brcmfmac: CONSOLE: 026816.727 wl0: link down (wl0.2)

This part at least is interesting and I would like to know how brcmfmac
deals with this order of events.

Regards,
Arend

2016-05-06 22:43:25

by Rafał Miłecki

[permalink] [raw]
Subject: Re: Problem with brcmfmac removing extra interface

On 23 March 2016 at 20:13, Rafał Miłecki <[email protected]> wrote:
> On 23 March 2016 at 11:44, Arend Van Spriel
> <[email protected]> wrote:
>> On 23-3-2016 9:47, Rafał Miłecki wrote:
>>> On 22 March 2016 at 21:24, Arend van Spriel
>>> <[email protected]> wrote:
>>>> On 22-3-2016 7:36, Rafał Miłecki wrote:
>>>>> Hi, any ideas / help regarding this issue?
>>>>
>>>> Restarting hostapd obviously is a valid scenario. My guess is we would
>>>> need to avoid interface removal in brcmf_notify_connect_status_ap. But
>>>> first I would like to know in which state the firmware is upon
>>>> brcmf_ap_add_vif. Can you provide a full log with FWCON debug level enabled?
>>>
>>> Sure, hope it helps!
>>
>> Ah. Actually would like to see some driver logging as well so
>> 'debug=0x101410'.
>
> Sure, there you go.

Did this allow you to understand this problem any better? Do you have
any idea for a solution?

--
Rafał

2016-05-25 07:56:30

by Rafał Miłecki

[permalink] [raw]
Subject: Re: Problem with brcmfmac removing extra interface

On 24 May 2016 at 21:49, Arend Van Spriel <[email protected]> wrote:
> Looking into the firmware code it turns out that the DOWN command brings
> down the entire stack iso single interface. This seems specific for some
> firmware branches so we have to find a better solution. Hante and I are
> looking into it.

I started working on brcmfmac few days ago and I realized that as
well. When having 4 AP interfaces and stopping hostapd nicely I get:

brcmfmac: [brcmf_fweh_handle_if_event] ifevent->action:3:BRCMF_E_IF_CHANGE
brcmfmac: action: 3 ifidx: 0 bsscfgidx: 0 flags: 0 role: 8

brcmfmac: [brcmf_fweh_handle_if_event] ifevent->action:2:BRCMF_E_IF_DEL
brcmfmac: action: 2 ifidx: 1 bsscfgidx: 2 flags: 0 role: 1
brcmfmac: brcmf_fweh_call_event_handler: no interface object

brcmfmac: [brcmf_fweh_handle_if_event] ifevent->action:2:BRCMF_E_IF_DEL
brcmfmac: action: 2 ifidx: 2 bsscfgidx: 3 flags: 0 role: 1
brcmfmac: brcmf_fweh_call_event_handler: no interface object

brcmfmac: [brcmf_fweh_handle_if_event] ifevent->action:2:BRCMF_E_IF_DEL
brcmfmac: action: 2 ifidx: 3 bsscfgidx: 4 flags: 0 role: 1
brcmfmac: brcmf_fweh_call_event_handler: no interface object

brcmfmac: [brcmf_fweh_handle_if_event] ifevent->action:1:BRCMF_E_IF_ADD
brcmfmac: action: 1 ifidx: 0 bsscfgidx: 0 flags: 0 role: 8
brcmfmac: adding wl0 (80:23:6a:a3:7d:95)

brcmfmac: [brcmf_fweh_handle_if_event] ifevent->action:3:BRCMF_E_IF_CHANGE
brcmfmac: action: 3 ifidx: 0 bsscfgidx: 0 flags: 0 role: 0

It means that all interfaces/BSS-es were dropped by firmware
internally and thanks to that we can create them again (start AP
again).

It doesn't happen if:
1) I simply add & remove interfaces manually with "iw"
2) I use BCM4366 and hostapd
In both above cases firmware keeps its internal interfaces/BSS-es and
I can't recreate interfaces.

--
Rafał

2016-05-24 19:49:41

by Arend Van Spriel

[permalink] [raw]
Subject: Re: Problem with brcmfmac removing extra interface

On 23-3-2016 20:13, Rafał Miłecki wrote:
> On 23 March 2016 at 11:44, Arend Van Spriel
> <[email protected]> wrote:
>> On 23-3-2016 9:47, Rafał Miłecki wrote:
>>> On 22 March 2016 at 21:24, Arend van Spriel
>>> <[email protected]> wrote:
>>>> On 22-3-2016 7:36, Rafał Miłecki wrote:
>>>>> Hi, any ideas / help regarding this issue?
>>>>
>>>> Restarting hostapd obviously is a valid scenario. My guess is
>>>> we would need to avoid interface removal in
>>>> brcmf_notify_connect_status_ap. But first I would like to know
>>>> in which state the firmware is upon brcmf_ap_add_vif. Can you
>>>> provide a full log with FWCON debug level enabled?
>>>
>>> Sure, hope it helps!
>>
>> Ah. Actually would like to see some driver logging as well so
>> 'debug=0x101410'.
>
> Sure, there you go.

A bit overdue, but here we go. Upon killing hostapd it results in
brcmf_cfg80211_stop_ap() for 'OpenWrt5x' bss and we give a DOWN command
for the interface to firmware. However, we get a link down for the
primary interface *and* second interface.

[ 172.799334] brcmfmac: brcmf_fil_cmd_int_set ifidx=1, cmd=3, value=1
[ 172.802111] brcmfmac: CONSOLE: 026767.523 wl0: link down (wl0)
[ 172.802147] brcmfmac: CONSOLE: 026767.523 wl0: link down (wl0.2)
[ 172.802166] brcmfmac: brcmf_fweh_event_worker event LINK (16) ifidx 0
bsscfg 0 addr 90:8d:78:66:3a:56
[ 172.802173] brcmfmac: brcmf_fweh_event_worker version 2 flags 0
status 0 reason 4
[ 172.802178] brcmutil: event payload, len=0
[ 172.802202] brcmfmac: brcmf_fweh_event_worker event LINK (16) ifidx 1
bsscfg 2 addr 92:8d:78:66:3a:57
[ 172.802208] brcmfmac: brcmf_fweh_event_worker version 2 flags 0
status 0 reason 4
[ 172.802212] brcmutil: event payload, len=0

Looking into the firmware code it turns out that the DOWN command brings
down the entire stack iso single interface. This seems specific for some
firmware branches so we have to find a better solution. Hante and I are
looking into it.

Regards,
Arend