2016-02-03 15:24:44

by Josh Boyer

[permalink] [raw]
Subject: ibss.c backtrace when batman-adv adds wireless interface

Hi All,

We've had a user report the backtrace below when loading batman-adv on
his machine. It looks like the cfg80211 layer is complaining about a
null bss returned, but I cannot tell if the rtlwifi driver or
batman-adv is in error here.

Thoughts?

josh

Jan 28 22:29:21 melissa.gathman.org kernel: batman_adv: B.A.T.M.A.N.
advanced 2015.2 (compatibility version 15) loaded
Jan 28 22:29:21 melissa.gathman.org kernel: wlp0s26u1u4: Selected IBSS
BSSID 02:1a:a0:3a:a6:ff based on configured SSID
Jan 28 22:29:21 melissa.gathman.org kernel: ------------[ cut here ]------------
Jan 28 22:29:21 melissa.gathman.org kernel: WARNING: CPU: 1 PID: 111
at net/wireless/ibss.c:35 __cfg80211_ibss_joined+0x166/0x190
[cfg80211]()
Jan 28 22:29:21 melissa.gathman.org kernel: Modules linked in:
batman_adv libcrc32c xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4
nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter
ip6t_REJECT nf_reject_ipv6 xt_conntrack tun ebtable_filter ebtable_nat
ebtable_broute bridge stp llc ebtables ip6table_mangle ip6table_nat
nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_raw
ip6table_security ip6table_filter ip6_tables iptable_mangle
iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat
nf_conntrack iptable_raw iptable_security arc4 rtl8192cu rtl_usb
rtl8192c_common rtlwifi mac80211 cfg80211 rfkill uvcvideo
videobuf2_vmalloc videobuf2_core videobuf2_memops v4l2_common
intel_rapl snd_hda_codec_hdmi snd_usb_audio joydev snd_usbmidi_lib
snd_hda_codec_realtek iosf_mbi x86_pkg_temp_thermal videodev coretemp
kvm_intel
Jan 28 22:29:21 melissa.gathman.org kernel: media snd_rawmidi cm109
snd_hda_codec_generic kvm snd_hda_intel iTCO_wdt snd_hda_codec
iTCO_vendor_support crct10dif_pclmul snd_hda_core ppdev snd_hwdep
snd_seq snd_seq_device snd_pcm crc32_pclmul crc32c_intel snd_timer snd
dcdbas lpc_ich mei_me parport_pc parport shpchp i2c_i801 mei tpm_tis
tpm soundcore nfsd auth_rpcgss nfs_acl lockd grace sunrpc
hid_microsoft i915 i2c_algo_bit drm_kms_helper drm e1000e serio_raw
ptp pps_core fjes video
Jan 28 22:29:21 melissa.gathman.org kernel: CPU: 1 PID: 111 Comm:
kworker/u32:5 Not tainted 4.3.3-301.fc23.x86_64 #1
Jan 28 22:29:21 melissa.gathman.org kernel: Hardware name: Dell Inc.
OptiPlex 790/0HY9JP, BIOS A06 07/25/2011
Jan 28 22:29:21 melissa.gathman.org kernel: Workqueue: cfg80211
cfg80211_event_work [cfg80211]
Jan 28 22:29:21 melissa.gathman.org kernel: 0000000000000000
000000008beeeb5f ffff8800368bfce0 ffffffff813a626f
Jan 28 22:29:21 melissa.gathman.org kernel: 0000000000000000
ffff8800368bfd18 ffffffff810a07c2 ffff8800c2bc8850
Jan 28 22:29:21 melissa.gathman.org kernel: ffff8800c2bc8000
ffff8801275bd558 0000000000000000 0000000000000286
Jan 28 22:29:21 melissa.gathman.org kernel: Call Trace:
Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff813a626f>]
dump_stack+0x44/0x55
Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810a07c2>]
warn_slowpath_common+0x82/0xc0
Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810a090a>]
warn_slowpath_null+0x1a/0x20
Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffffa05dfcd6>]
__cfg80211_ibss_joined+0x166/0x190 [cfg80211]
Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffffa05bb02c>]
cfg80211_process_wdev_events+0x8c/0x190 [cfg80211]
Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffffa05bb162>]
cfg80211_process_rdev_events+0x32/0x70 [cfg80211]
Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffffa05b601e>]
cfg80211_event_work+0x1e/0x30 [cfg80211]
Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810b8cde>]
process_one_work+0x19e/0x3f0
Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810b8f7e>]
worker_thread+0x4e/0x450
Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810b8f30>] ?
process_one_work+0x3f0/0x3f0
Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810bede8>]
kthread+0xd8/0xf0
Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810bed10>] ?
kthread_worker_fn+0x160/0x160
Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff8178195f>]
ret_from_fork+0x3f/0x70
Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810bed10>] ?
kthread_worker_fn+0x160/0x160
Jan 28 22:29:21 melissa.gathman.org kernel: ---[ end trace 04ee7067960ce525 ]---
Jan 28 22:29:21 melissa.gathman.org kernel: batman_adv: bat0: Adding
interface: wlp0s26u1u4


2016-02-03 15:41:41

by Johannes Berg

[permalink] [raw]
Subject: Re: ibss.c backtrace when batman-adv adds wireless interface

On Wed, 2016-02-03 at 10:26 -0500, Josh Boyer wrote:
> On Wed, Feb 3, 2016 at 10:24 AM, Josh Boyer <[email protected]
> g> wrote:
> > Hi All,
> >
> > We've had a user report the backtrace below when loading batman-adv
> > on
> > his machine.  It looks like the cfg80211 layer is complaining about
> > a
> > null bss returned, but I cannot tell if the rtlwifi driver or
> > batman-adv is in error here.
> >
> > Thoughts?
>
> Sorry, forgot to include the link to the actual bug report:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1304428
>
> Reporter says this is new with 4.3.y and did not happen on e.g.
> 4.2.8.
>

AFAICT this should be a driver (or perhaps mac80211) issue, but I don't
see any information about the driver used.

johannes

2016-02-03 17:14:57

by Sven Eckelmann

[permalink] [raw]
Subject: Re: [B.A.T.M.A.N.] ibss.c backtrace when batman-adv adds wireless interface

On Wednesday 03 February 2016 10:24:43 Josh Boyer wrote:
> Hi All,
>
> We've had a user report the backtrace below when loading batman-adv on
> his machine. It looks like the cfg80211 layer is complaining about a
> null bss returned, but I cannot tell if the rtlwifi driver or
> batman-adv is in error here.
>
> Thoughts?

About the attachment "batman-adv oopses with kernel-4.3.4-300+debug". Sorry,
this was my fault and caused by an copy+paster error while adding some
lockdep_asserts. This caused too many lockdep reports. It was fixed a while
ago upstream in 008a37448707 ("batman-adv: Fix lockdep annotation of
batadv_tlv_container_remove").

Kind regards,
Sven


Attachments:
signature.asc (819.00 B)
This is a digitally signed message part.

2016-02-03 17:35:32

by Sven Eckelmann

[permalink] [raw]
Subject: Re: [B.A.T.M.A.N.] ibss.c backtrace when batman-adv adds wireless interface

On Wednesday 03 February 2016 18:09:40 Sven Eckelmann wrote:
> > AFAICT this should be a driver (or perhaps mac80211) issue, but I don't
> > see any information about the driver used.
>
> It doesn't really look like it is batman-adv specific. batman-adv also doesn't
> be active because the message "batman_adv: bat0: Adding interface: " is
> missing.

Maybe I should clarify a little bit. I meant that that message is missing
before the problem happens. The interface message appears after the problem
was detected.

And maybe I should also inform you that many batman-adv refcounting
patches [1] for some long standing problems are getting queued up by the brave
linux-stable people. But there are also two patches [2,3] which Antonio hasn't
submitted yet.

Kind regards,
Sven

[1] https://kernel.org/pub/linux/kernel/v4.x/stable-review/patch-4.3.5-rc1.gz
[2] https://lists.open-mesh.org/pipermail/b.a.t.m.a.n/2016-January/014304.html
[3] https://lists.open-mesh.org/pipermail/b.a.t.m.a.n/2016-January/014305.html


Attachments:
signature.asc (819.00 B)
This is a digitally signed message part.

2016-02-03 17:09:43

by Sven Eckelmann

[permalink] [raw]
Subject: Re: [B.A.T.M.A.N.] ibss.c backtrace when batman-adv adds wireless interface

On Wednesday 03 February 2016 16:41:30 Johannes Berg wrote:
> On Wed, 2016-02-03 at 10:26 -0500, Josh Boyer wrote:
> > On Wed, Feb 3, 2016 at 10:24 AM, Josh Boyer <[email protected]
> > g> wrote:
> > > Hi All,
> > >
> > > We've had a user report the backtrace below when loading batman-adv
> > > on
> > > his machine. It looks like the cfg80211 layer is complaining about
> > > a
> > > null bss returned, but I cannot tell if the rtlwifi driver or
> > > batman-adv is in error here.
> > >
> > > Thoughts?
> >
> > Sorry, forgot to include the link to the actual bug report:
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=1304428
> >
> > Reporter says this is new with 4.3.y and did not happen on e.g.
> > 4.2.8.
> >
>
> AFAICT this should be a driver (or perhaps mac80211) issue, but I don't
> see any information about the driver used.

It doesn't really look like it is batman-adv specific. batman-adv also doesn't
be active because the message "batman_adv: bat0: Adding interface: " is
missing.

batman-adv is also not responsible for the underlying device. So it will not
join IBSS and thus should not trigger the "wlp0s26u1u4: Selected IBSS BSSID"
seen in the log.

But that said, I already saw weirder things happen because of memory
corruptions. So I will not say that batman-adv cannot be the culprit but right
now I don't see how.

The user can also try (when he thinks that batman-adv is to blame) to bisect
the batman-adv versions via the git of the out-of-tree module [1]. The version [2]
from 4.1 should be roughly v2015.0 (b6ff92aa334083dee8a1f5f0c5aa0f7c4c2584cf),
v4.2 should be v2015.1 (95b64b1cef38340f48f4eba99689e0fb1feddcd0) and v4.3 should
not be newer than v2015.2 (e25bd38b0f8b4aeca85ef24884fa5eadd6270709). So bisecting
between b6ff92aa334083dee8a1f5f0c5aa0f7c4c2584cf and
e25bd38b0f8b4aeca85ef24884fa5eadd6270709 (~6 steps) should cover it. I personally
would recommend to start on 4.2.8 with v2015.2
(e25bd38b0f8b4aeca85ef24884fa5eadd6270709)

Kind regards,
Sven

[1] https://git.open-mesh.org/batman-adv.git
[2] https://www.open-mesh.org/projects/open-mesh/wiki/Download#batman-adv-in-the-Linux-tree


Attachments:
signature.asc (819.00 B)
This is a digitally signed message part.

2016-02-03 15:49:15

by Josh Boyer

[permalink] [raw]
Subject: Re: ibss.c backtrace when batman-adv adds wireless interface

On Wed, Feb 3, 2016 at 10:41 AM, Johannes Berg
<[email protected]> wrote:
> On Wed, 2016-02-03 at 10:26 -0500, Josh Boyer wrote:
>> On Wed, Feb 3, 2016 at 10:24 AM, Josh Boyer <[email protected]
>> g> wrote:
>> > Hi All,
>> >
>> > We've had a user report the backtrace below when loading batman-adv
>> > on
>> > his machine. It looks like the cfg80211 layer is complaining about
>> > a
>> > null bss returned, but I cannot tell if the rtlwifi driver or
>> > batman-adv is in error here.
>> >
>> > Thoughts?
>>
>> Sorry, forgot to include the link to the actual bug report:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1304428
>>
>> Reporter says this is new with 4.3.y and did not happen on e.g.
>> 4.2.8.
>>
>
> AFAICT this should be a driver (or perhaps mac80211) issue, but I don't
> see any information about the driver used.

The backtrace has all the modules loaded included in it. rtlwifi is
listed there and it's the only wireless driver in use (rtl8192cu
specifically via USB).

josh

2016-02-03 15:26:51

by Josh Boyer

[permalink] [raw]
Subject: Re: ibss.c backtrace when batman-adv adds wireless interface

On Wed, Feb 3, 2016 at 10:24 AM, Josh Boyer <[email protected]> wrote:
> Hi All,
>
> We've had a user report the backtrace below when loading batman-adv on
> his machine. It looks like the cfg80211 layer is complaining about a
> null bss returned, but I cannot tell if the rtlwifi driver or
> batman-adv is in error here.
>
> Thoughts?

Sorry, forgot to include the link to the actual bug report:

https://bugzilla.redhat.com/show_bug.cgi?id=1304428

Reporter says this is new with 4.3.y and did not happen on e.g. 4.2.8.


josh

>
> Jan 28 22:29:21 melissa.gathman.org kernel: batman_adv: B.A.T.M.A.N.
> advanced 2015.2 (compatibility version 15) loaded
> Jan 28 22:29:21 melissa.gathman.org kernel: wlp0s26u1u4: Selected IBSS
> BSSID 02:1a:a0:3a:a6:ff based on configured SSID
> Jan 28 22:29:21 melissa.gathman.org kernel: ------------[ cut here ]------------
> Jan 28 22:29:21 melissa.gathman.org kernel: WARNING: CPU: 1 PID: 111
> at net/wireless/ibss.c:35 __cfg80211_ibss_joined+0x166/0x190
> [cfg80211]()
> Jan 28 22:29:21 melissa.gathman.org kernel: Modules linked in:
> batman_adv libcrc32c xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4
> nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter
> ip6t_REJECT nf_reject_ipv6 xt_conntrack tun ebtable_filter ebtable_nat
> ebtable_broute bridge stp llc ebtables ip6table_mangle ip6table_nat
> nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_raw
> ip6table_security ip6table_filter ip6_tables iptable_mangle
> iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat
> nf_conntrack iptable_raw iptable_security arc4 rtl8192cu rtl_usb
> rtl8192c_common rtlwifi mac80211 cfg80211 rfkill uvcvideo
> videobuf2_vmalloc videobuf2_core videobuf2_memops v4l2_common
> intel_rapl snd_hda_codec_hdmi snd_usb_audio joydev snd_usbmidi_lib
> snd_hda_codec_realtek iosf_mbi x86_pkg_temp_thermal videodev coretemp
> kvm_intel
> Jan 28 22:29:21 melissa.gathman.org kernel: media snd_rawmidi cm109
> snd_hda_codec_generic kvm snd_hda_intel iTCO_wdt snd_hda_codec
> iTCO_vendor_support crct10dif_pclmul snd_hda_core ppdev snd_hwdep
> snd_seq snd_seq_device snd_pcm crc32_pclmul crc32c_intel snd_timer snd
> dcdbas lpc_ich mei_me parport_pc parport shpchp i2c_i801 mei tpm_tis
> tpm soundcore nfsd auth_rpcgss nfs_acl lockd grace sunrpc
> hid_microsoft i915 i2c_algo_bit drm_kms_helper drm e1000e serio_raw
> ptp pps_core fjes video
> Jan 28 22:29:21 melissa.gathman.org kernel: CPU: 1 PID: 111 Comm:
> kworker/u32:5 Not tainted 4.3.3-301.fc23.x86_64 #1
> Jan 28 22:29:21 melissa.gathman.org kernel: Hardware name: Dell Inc.
> OptiPlex 790/0HY9JP, BIOS A06 07/25/2011
> Jan 28 22:29:21 melissa.gathman.org kernel: Workqueue: cfg80211
> cfg80211_event_work [cfg80211]
> Jan 28 22:29:21 melissa.gathman.org kernel: 0000000000000000
> 000000008beeeb5f ffff8800368bfce0 ffffffff813a626f
> Jan 28 22:29:21 melissa.gathman.org kernel: 0000000000000000
> ffff8800368bfd18 ffffffff810a07c2 ffff8800c2bc8850
> Jan 28 22:29:21 melissa.gathman.org kernel: ffff8800c2bc8000
> ffff8801275bd558 0000000000000000 0000000000000286
> Jan 28 22:29:21 melissa.gathman.org kernel: Call Trace:
> Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff813a626f>]
> dump_stack+0x44/0x55
> Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810a07c2>]
> warn_slowpath_common+0x82/0xc0
> Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810a090a>]
> warn_slowpath_null+0x1a/0x20
> Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffffa05dfcd6>]
> __cfg80211_ibss_joined+0x166/0x190 [cfg80211]
> Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffffa05bb02c>]
> cfg80211_process_wdev_events+0x8c/0x190 [cfg80211]
> Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffffa05bb162>]
> cfg80211_process_rdev_events+0x32/0x70 [cfg80211]
> Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffffa05b601e>]
> cfg80211_event_work+0x1e/0x30 [cfg80211]
> Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810b8cde>]
> process_one_work+0x19e/0x3f0
> Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810b8f7e>]
> worker_thread+0x4e/0x450
> Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810b8f30>] ?
> process_one_work+0x3f0/0x3f0
> Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810bede8>]
> kthread+0xd8/0xf0
> Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810bed10>] ?
> kthread_worker_fn+0x160/0x160
> Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff8178195f>]
> ret_from_fork+0x3f/0x70
> Jan 28 22:29:21 melissa.gathman.org kernel: [<ffffffff810bed10>] ?
> kthread_worker_fn+0x160/0x160
> Jan 28 22:29:21 melissa.gathman.org kernel: ---[ end trace 04ee7067960ce525 ]---
> Jan 28 22:29:21 melissa.gathman.org kernel: batman_adv: bat0: Adding
> interface: wlp0s26u1u4