I receive a kernel panic with rt2500pci built-in, but if built as a
module, I can insmod it without problem. I don't have a serial
console, and I tried to capture the panic with netconsole (using a
different interface of course) but the panic happens immediately after
rt2500pci is initialized, or in other words, too soon for netconsole.
I'd rather not copy over everything unless necessary. Here's the first portion:
ACPI: PCI Interrupt 0000:02:0a.0[A] -> GSI 22 (level, low) -> IRQ 18
phy0 -> rt2x00_set_chip: Info - Chipset detected - rt: 0201, rf: 0003,
rev: 00000004
WARNING: at lib/kref.c:33 kref_get()
[<c0229da9>] kref_get+0x3d/0x44
On Sat, 2007-09-08 at 00:40 -0400, Rob Hussey wrote:
> You don't have to answer that. Going through the dmesg it seems a lot
> of devices rightfully have null for parent.
Ok so that means I read the stackdumps wrong.
> The relevant line seems to
> be:
> [ 28.348658] kobject phy0: registering. parent: ieee80211, set: devices
That's interesting though. I'll have to take another look. Maybe somehow
we initialise cfg80211 too late. In fact.. Hmm. Can you try something?
When having rt2500pci built in obviously you also have cfg80211 built
in. Can you go to the end of the file net/wireless/core.c and exchange
the module_init(cfg80211_init) with subsys_initcall(cfg80211_init)? It
looks like that might be the problem.
johannes
Forgot to "reply to all." Resending.
On 9/7/07, Johannes Berg <[email protected]> wrote:
> On Fri, 2007-09-07 at 15:31 -0400, Rob Hussey wrote:
>
> > This was with wireless-dev git. I can try DEBUG_KOBJECT once I get a chance.
>
> That'd be good.
>
I took some pictures of the screen with DEBUG_KOBJECT enabled. It's
tough to get it all though, since the messages don't really fit on one
screen. I played around with the vga= line, and was able to get the
scrolling to stop at different points (the stack trace and all appears
about 1 msec after booting, so this was my only choice). The pictures
can be found here:
http://www.healthcarelinen.com/misc/
These two are probably the best:
http://www.healthcarelinen.com/misc/101_1988.jpg
http://www.healthcarelinen.com/misc/101_1992.jpg
Hopefully the pictures have what you need.
> > I'd like to try to bisect this though, but I'm having trouble finding
> > a point in which it worked.
>
> > If I do:
> > git clone git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-dev.git
> > and then:
> > git pull . remotes/origin/rt2x00
> >
> > Then my commit history has all rt2x00 commits on top, and trying to
> > reset to an earlier point in time reverts rt2x00 commits, but not
> > wireless-dev commits (unless I reset to before the initial rt2x00
> > commit). This causes compile errors mostly involving undeclared stuff
> > prefixed "IEEE80211_" Is there a better way to do this? Did I even
> > start out correctly?
>
> I'm not sure you can bisect due to the way mac80211/rt2x00 are currently
> developed, in lockstep but in different branches that are pulled
> together.
>
Bummer.
On 9/10/07, Rob Hussey <[email protected]> wrote:
> I had to change the module_init() to subsys_initcall() in ieee80211.c
> and rc80211_simple.c, as well as the original change in
> net/wireless/core.c.
After more testing, it seems to also work just as well if ieee80211.c
is left with the module_init() and both rc80211_simple.c and
net/wireless/core.c are switched to subsys_initcall().
On Saturday 08 September 2007, Rob Hussey wrote:
> On 9/8/07, Ivo van Doorn <[email protected]> wrote:
> >
> > And there was no rt2x00lib, rt2x00pci or rt2500pci error message
> > in the log that could indicate why the device is not present?
>
> Not in the log, but on the screen. I should've noticed not everything
> was in there. Here's what shows up:
>
> wmaster0: Failed to select rate control algorithm
Did you load rc80211_simple or rc80211_lowest after modprobing mac80211?
Ivo
On Wed, 2007-09-05 at 23:25 -0400, Rob Hussey wrote:
> > ACPI: PCI Interrupt 0000:02:0a.0[A] -> GSI 22 (level, low) -> IRQ 18
> > phy0 -> rt2x00_set_chip: Info - Chipset detected - rt: 0201, rf: 0003,
> > rev: 00000004
> > WARNING: at lib/kref.c:33 kref_get()
> > [<c0229da9>] kref_get+0x3d/0x44
> kobject_get
> kobject_shadow
> kobject_set_name
> device_add
> wiphy_register
I have to admit I don't understand this. It seems that device_add is
called with an invalid device, but that's very odd because
device_initialise must have been called before.
johannes
Also, config can be found here:
http://www.healthcarelinen.com/misc/.config
On 9/5/07, Rob Hussey <[email protected]> wrote:
> Sorry hit enter before I was done typing, message continues:
>
> On 9/5/07, Rob Hussey <[email protected]> wrote:
> > I receive a kernel panic with rt2500pci built-in, but if built as a
> > module, I can insmod it without problem. I don't have a serial
> > console, and I tried to capture the panic with netconsole (using a
> > different interface of course) but the panic happens immediately after
> > rt2500pci is initialized, or in other words, too soon for netconsole.
> >
> > I'd rather not copy over everything unless necessary. Here's the first portion:
> >
> > ACPI: PCI Interrupt 0000:02:0a.0[A] -> GSI 22 (level, low) -> IRQ 18
> > phy0 -> rt2x00_set_chip: Info - Chipset detected - rt: 0201, rf: 0003,
> > rev: 00000004
> > WARNING: at lib/kref.c:33 kref_get()
> > [<c0229da9>] kref_get+0x3d/0x44
> kobject_get
> kobject_shadow
> kobject_set_name
> device_add
> wiphy_register
> ieee80211_register_hw
> rt2x00lib_probe_dev
> rt2x00pci_probe
> pci_match_device
> pci_device_probe
> driver_probe_device
> klist_next
> __driver_attach
> bus_for_each_dev
> driver_attach
> __driver_attach
> bus_add_driver
> __pci_register_driver
> kernel_init
>
> ... can't type everything... skipping
>
> BUG: unable to handle kernel NULL pointer dereference at virtual
> address 00000024
> printing eip:
> c01a9966
> *pde = 00000000
> Oops: 0000 [#1]
> PREEMPT SMP
> Modules linked in:
> CPU: 1
> EIP: 0060:[<c01a9966>] Not tainted VLI
> EIP is at sysfs_addrm_start+0x21/0x87
>
> skipping...
>
> Call Trace:
> create_dir
> sysfs_create_dir
> kobject_get
> kobject_shadow_add
> kobject_set_name
> device_add
> wiphy_register
> ieee80211_register_hw
> rt2x00lib_prove_dev
> rt2x00pci_probe
> pci_match_device
> pci_device_probe
> driver_probe_device
> klist_next
> __driver_attach
> bus_for_each_dev
> driver_attach
> __driver_attach
> bus_add_driver
> __pci_register_driver
> kernel_init
> schedule_tail
> ret_from_fork
> kernel_init
> kernel_init
> kernel_thread_helper
>
> ...
>
> Kernel panic - not syncing: Attempting to kill init!
>
>
> I likely left out important parts, if so, just tell me and I'll try to
> post them.
>
On 9/6/07, Johannes Berg <[email protected]> wrote:
>
> I have no idea what's going on here. Can you enable DEBUG_KOBJECT? Oh
> also, what kernel is this with?
>
This was with wireless-dev git. I can try DEBUG_KOBJECT once I get a chance.
I'd like to try to bisect this though, but I'm having trouble finding
a point in which it worked.
If I do:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-dev.git
and then:
git pull . remotes/origin/rt2x00
Then my commit history has all rt2x00 commits on top, and trying to
reset to an earlier point in time reverts rt2x00 commits, but not
wireless-dev commits (unless I reset to before the initial rt2x00
commit). This causes compile errors mostly involving undeclared stuff
prefixed "IEEE80211_" Is there a better way to do this? Did I even
start out correctly?
Thanks,
Rob
On Saturday 08 September 2007, Rob Hussey wrote:
> On 9/8/07, Johannes Berg <[email protected]> wrote:
> >
> > ?! That shouldn't have an effect like that. Are you positive that you
> > put the register_hw call back?
>
> Yeah, I thought the same thing, so I even did a git-reset --hard HEAD
> and double-checked rt2x00dev.c to make sure everything was back to
> normal before editing core.c again.
And there was no rt2x00lib, rt2x00pci or rt2500pci error message
in the log that could indicate why the device is not present?
Could you compile rt2x00 with debug enabled and see what additional
messages are being printed out?
Ivo
On Fri, 2007-09-07 at 17:38 -0400, Rob Hussey wrote:
> I took some pictures of the screen with DEBUG_KOBJECT enabled. It's
> tough to get it all though, since the messages don't really fit on one
> screen. I played around with the vga= line, and was able to get the
> scrolling to stop at different points (the stack trace and all appears
> about 1 msec after booting, so this was my only choice). The pictures
> can be found here:
> http://www.healthcarelinen.com/misc/
>
> These two are probably the best:
> http://www.healthcarelinen.com/misc/101_1988.jpg
> http://www.healthcarelinen.com/misc/101_1992.jpg
>
> Hopefully the pictures have what you need.
The line
kobject phy0: registering. parent: <NULL>, set: devices
points out what I thought was the problem. However, I don't understand
why it happens.
In rt2500pci we see:
rt2500pci_probe_hw_mode:
SET_IEEE80211_DEV(rt2x00dev->hw, &rt2x00dev_pci(rt2x00dev)->dev);
which expands to
set_wiphy_dev(hw->wiphy, dev);
which expands to
wiphy->dev.parent = dev;
it is called from rt2500pci_probe_hw which in turn is called from
rt2x00lib_probe_dev, which later calls rt2x00lib_probe_hw which is what
calls ieee80211_register_hw which is the path leading to the bug.
But there's no way &rt2x00dev_pci(rt2x00dev)->dev can be NULL! Sorry, I
don't see right now how this can happen. Anybody have a better idea?
johannes
On 9/8/07, Johannes Berg <[email protected]> wrote:
>
> Can you go to the end of the file net/wireless/core.c and exchange
> the module_init(cfg80211_init) with subsys_initcall(cfg80211_init)? It
> looks like that might be the problem.
Much like when I commented out the ieee80211_register_hw call, the
kernel booted fine, but there was no sign of the wireless card in the
logs or otherwise.
On Saturday 08 September 2007, Rob Hussey wrote:
> On 9/8/07, Ivo van Doorn <[email protected]> wrote:
> >
> > Did you load rc80211_simple or rc80211_lowest after modprobing mac80211?
>
> Neither. These messages were with everything built in (including
> mac80211) along with the change Johannes suggested, which was to
> exchange module_init(cfg80211_init) with
> subsys_initcall(cfg80211_init) in net/wireless/core.c
>
> Even when it's built as modules, I've never had to load anything. It's
> always just automatically selected the simple algorithm.
Well for some reason mac80211 cannot find a correct rate selection algorithm,
that stops the initialization of rt2x00 and in turn that cause that your interface
does not show up.
Ivo
On Wed, 2007-09-05 at 23:25 -0400, Rob Hussey wrote:
> > ACPI: PCI Interrupt 0000:02:0a.0[A] -> GSI 22 (level, low) -> IRQ 18
> > phy0 -> rt2x00_set_chip: Info - Chipset detected - rt: 0201, rf: 0003,
> > rev: 00000004
> > WARNING: at lib/kref.c:33 kref_get()
> > [<c0229da9>] kref_get+0x3d/0x44
> kobject_get
> kobject_shadow
> kobject_set_name
> device_add
> wiphy_register
Ok, this comes from
WARN_ON(!atomic_read(&kref->refcount));
which means that somehow the refcount is zero.
> BUG: unable to handle kernel NULL pointer dereference at virtual
> address 00000024
> EIP is at sysfs_addrm_start+0x21/0x87
> Call Trace:
> create_dir
> sysfs_create_dir
> kobject_get
> kobject_shadow_add
> kobject_set_name
> device_add
> wiphy_register
I wish x86 had proper backtraces. What are kobject_get and
kobject_set_name doing in there??? Can't people get proper machines?
powerpc has really nice backtraces... Anyhow, that appears to be sysfs
trying to access kobj->parent which appears to be NULL.
I have no idea what's going on here. Can you enable DEBUG_KOBJECT? Oh
also, what kernel is this with?
johannes
On 9/8/07, Johannes Berg <[email protected]> wrote:
>
> ?! That shouldn't have an effect like that. Are you positive that you
> put the register_hw call back?
Yeah, I thought the same thing, so I even did a git-reset --hard HEAD
and double-checked rt2x00dev.c to make sure everything was back to
normal before editing core.c again.
On Sat, 2007-09-08 at 04:26 -0400, Rob Hussey wrote:
> Much like when I commented out the ieee80211_register_hw call, the
> kernel booted fine, but there was no sign of the wireless card in the
> logs or otherwise.
?! That shouldn't have an effect like that. Are you positive that you
put the register_hw call back?
johannes
It seems that rc80211_simple is being built-in. I don't know why
mac80211 can't find it, and I have no idea where to go from here.
rob@rob-desktop:~/Source/wireless-dev$ grep -R "rc80211_simple" *
net/mac80211/Makefile:obj-$(CONFIG_MAC80211) += mac80211.o rc80211_simple.o
net/mac80211/.built-in.o.cmd:cmd_net/mac80211/built-in.o := ld -m
elf_i386 -m elf_i386 -r -o net/mac80211/built-in.o
net/mac80211/mac80211.o net/mac80211/rc80211_simple.o
Binary file net/mac80211/rc80211_simple.o matches
net/mac80211/.rc80211_simple.o.cmd:cmd_net/mac80211/rc80211_simple.o
:= gcc -m32 -Wp,-MD,net/mac80211/.rc80211_simple.o.d -nostdinc
-isystem /usr/lib/gcc/i486-linux-gnu/4.1.2/include -D__KERNEL__
-Iinclude -include include/linux/autoconf.h -Wall -Wundef
-Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common
-Werror-implicit-function-declaration -O2 -pipe -msoft-float
-mregparm=3 -freg-struct-return -mpreferred-stack-boundary=2
-march=i686 -mtune=pentium4 -ffreestanding -maccumulate-outgoing-args
-Iinclude/asm-i386/mach-default -fomit-frame-pointer
-fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign
-D"KBUILD_STR(s)=\#s" -D"KBUILD_BASENAME=KBUILD_STR(rc80211_simple)"
-D"KBUILD_MODNAME=KBUILD_STR(rc80211_simple)" -c -o
net/mac80211/.tmp_rc80211_simple.o net/mac80211/rc80211_simple.c
net/mac80211/.rc80211_simple.o.cmd:deps_net/mac80211/rc80211_simple.o := \
net/mac80211/.rc80211_simple.o.cmd: net/mac80211/rc80211_simple.c \
net/mac80211/.rc80211_simple.o.cmd:net/mac80211/rc80211_simple.o:
$(deps_net/mac80211/rc80211_simple.o)
net/mac80211/.rc80211_simple.o.cmd:$(deps_net/mac80211/rc80211_simple.o):
Binary file net/mac80211/built-in.o matches
Binary file net/built-in.o matches
Binary file vmlinux matches
Binary file vmlinux.o matches
On 9/8/07, Rob Hussey <[email protected]> wrote:
> Not sure if it's any help, but I've attached dmesg output with rt2x00
> built as modules. DEBUG_KOBJECT is still enabled, so maybe there's
> some sorta clue in there. BTW, if I do:
> grep -c "parent: <NULL>" dmesg.txt
> I get: 32
> Is that normal?
>
You don't have to answer that. Going through the dmesg it seems a lot
of devices rightfully have null for parent. The relevant line seems to
be:
[ 28.348658] kobject phy0: registering. parent: ieee80211, set: devices
Strange that it correctly registers as a module but not built-in.
On 9/8/07, Ivo van Doorn <[email protected]> wrote:
>
> Did you load rc80211_simple or rc80211_lowest after modprobing mac80211?
Neither. These messages were with everything built in (including
mac80211) along with the change Johannes suggested, which was to
exchange module_init(cfg80211_init) with
subsys_initcall(cfg80211_init) in net/wireless/core.c
Even when it's built as modules, I've never had to load anything. It's
always just automatically selected the simple algorithm.
On Sat, 2007-09-08 at 12:39 -0400, Rob Hussey wrote:
> Not in the log, but on the screen. I should've noticed not everything
> was in there. Here's what shows up:
>
> wmaster0: Failed to select rate control algorithm
> phy0 -> rt2x00lib_probe_dev: Error - Failed to initialize hw.
Ok, then the change I proposed to you definitely fixed the bug, but this
is an unrelated one. I'll have to look why it fails to init the rate
control when that's built in.
johannes
On Fri, 2007-09-07 at 15:31 -0400, Rob Hussey wrote:
> This was with wireless-dev git. I can try DEBUG_KOBJECT once I get a chance.
That'd be good.
> I'd like to try to bisect this though, but I'm having trouble finding
> a point in which it worked.
> If I do:
> git clone git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-dev.git
> and then:
> git pull . remotes/origin/rt2x00
>
> Then my commit history has all rt2x00 commits on top, and trying to
> reset to an earlier point in time reverts rt2x00 commits, but not
> wireless-dev commits (unless I reset to before the initial rt2x00
> commit). This causes compile errors mostly involving undeclared stuff
> prefixed "IEEE80211_" Is there a better way to do this? Did I even
> start out correctly?
I'm not sure you can bisect due to the way mac80211/rt2x00 are currently
developed, in lockstep but in different branches that are pulled
together.
johannes
On Mon, 2007-09-10 at 12:31 +0200, Johannes Berg wrote:
> Ok, then the change I proposed to you definitely fixed the bug, but this
> is an unrelated one. I'll have to look why it fails to init the rate
> control when that's built in.
Ah, I see, same bug.
mac80211 initialises with module_init(ieee80211_init); and the rate
control algorithms do the same, this means they haven't been initialised
yet when your device is registered.
Can you change all module_init() in net/mac80211/ to subsys_initcall()
as well?
johannes
Not sure if it's any help, but I've attached dmesg output with rt2x00
built as modules. DEBUG_KOBJECT is still enabled, so maybe there's
some sorta clue in there. BTW, if I do:
grep -c "parent: <NULL>" dmesg.txt
I get: 32
Is that normal?
Also, I commented out the ieee80211_register_hw call in
rt2x00lib_probe_hw, just to check, and the kernel did boot correctly,
but of course my wireless card didn't work at all.
Sorry hit enter before I was done typing, message continues:
On 9/5/07, Rob Hussey <[email protected]> wrote:
> I receive a kernel panic with rt2500pci built-in, but if built as a
> module, I can insmod it without problem. I don't have a serial
> console, and I tried to capture the panic with netconsole (using a
> different interface of course) but the panic happens immediately after
> rt2500pci is initialized, or in other words, too soon for netconsole.
>
> I'd rather not copy over everything unless necessary. Here's the first portion:
>
> ACPI: PCI Interrupt 0000:02:0a.0[A] -> GSI 22 (level, low) -> IRQ 18
> phy0 -> rt2x00_set_chip: Info - Chipset detected - rt: 0201, rf: 0003,
> rev: 00000004
> WARNING: at lib/kref.c:33 kref_get()
> [<c0229da9>] kref_get+0x3d/0x44
kobject_get
kobject_shadow
kobject_set_name
device_add
wiphy_register
ieee80211_register_hw
rt2x00lib_probe_dev
rt2x00pci_probe
pci_match_device
pci_device_probe
driver_probe_device
klist_next
__driver_attach
bus_for_each_dev
driver_attach
__driver_attach
bus_add_driver
__pci_register_driver
kernel_init
... can't type everything... skipping
BUG: unable to handle kernel NULL pointer dereference at virtual
address 00000024
printing eip:
c01a9966
*pde = 00000000
Oops: 0000 [#1]
PREEMPT SMP
Modules linked in:
CPU: 1
EIP: 0060:[<c01a9966>] Not tainted VLI
EIP is at sysfs_addrm_start+0x21/0x87
skipping...
Call Trace:
create_dir
sysfs_create_dir
kobject_get
kobject_shadow_add
kobject_set_name
device_add
wiphy_register
ieee80211_register_hw
rt2x00lib_prove_dev
rt2x00pci_probe
pci_match_device
pci_device_probe
driver_probe_device
klist_next
__driver_attach
bus_for_each_dev
driver_attach
__driver_attach
bus_add_driver
__pci_register_driver
kernel_init
schedule_tail
ret_from_fork
kernel_init
kernel_init
kernel_thread_helper
...
Kernel panic - not syncing: Attempting to kill init!
I likely left out important parts, if so, just tell me and I'll try to
post them.
On 9/8/07, Ivo van Doorn <[email protected]> wrote:
>
> And there was no rt2x00lib, rt2x00pci or rt2500pci error message
> in the log that could indicate why the device is not present?
Not in the log, but on the screen. I should've noticed not everything
was in there. Here's what shows up:
wmaster0: Failed to select rate control algorithm
phy0 -> rt2x00lib_probe_dev: Error - Failed to initialize hw.
ACPI: PCI interrupt for device 0000:02:0a.0 disabled
rt2500pci: probe of 0000:02:0a.0 failed with error -2
> Could you compile rt2x00 with debug enabled and see what additional
> messages are being printed out?
The above is with debug enabled, and I yesterday I attached dmesg
output with rt2x00 built as modules and debug enabled.
On Mon, 2007-09-10 at 17:25 -0400, Rob Hussey wrote:
> On 9/10/07, Rob Hussey <[email protected]> wrote:
> > I had to change the module_init() to subsys_initcall() in ieee80211.c
> > and rc80211_simple.c, as well as the original change in
> > net/wireless/core.c.
>
> After more testing, it seems to also work just as well if ieee80211.c
> is left with the module_init() and both rc80211_simple.c and
> net/wireless/core.c are switched to subsys_initcall().
Yeah, but that might leave mac80211 in a bit of a messy state, we really
should make sure that it is initialised before devices are registered on
it.
johannes
On 9/10/07, Johannes Berg <[email protected]> wrote:
>
> Can you change all module_init() in net/mac80211/ to subsys_initcall()
> as well?
Thank you Johannes! This worked very well. In fact, I'm emailing this
message using my built-in rt2x00 driver.
I had to change the module_init() to subsys_initcall() in ieee80211.c
and rc80211_simple.c, as well as the original change in
net/wireless/core.c. After that, it seems to work well either built-in
or as modules.
