Here is a report about lockups and/or(?) oopses while HALD is obviously
traversing sysfs files of FireWire devices, seen under Fedora Core
kernels 2.6.{16,17,18} on an SMP setup:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=188140
> May 17 11:49:42 malbec kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000020
> May 17 11:49:42 malbec kernel: printing eip:
> May 17 11:49:42 malbec kernel: c019797d
> May 17 11:49:42 malbec kernel: *pde = 14924001
> May 17 11:49:42 malbec kernel: Oops: 0000 [#1]
> May 17 11:49:42 malbec kernel: SMP
> May 17 11:49:42 malbec kernel: last sysfs file: /class/net/eth0/address
> May 17 11:49:42 malbec kernel: Modules linked in: sunrpc xt_limit xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables ip6table_filter ip6_tables x_tables ipv6 dm_multipath lp parport_pc parport floppy nvram ohci1394 ieee1394 uhci_hcd sg ohci_hcd ehci_hcd snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_emu10k1 snd_ac97_codec snd_ac97_bus snd_usb_audio snd_usb_lib snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event i2c_piix4 snd_seq i2c_core snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_device snd_timer snd_page_alloc snd_util_mem emu10k1_gp snd_hwdep gameport snd soundcore e100 mii dm_snapshot dm_zero dm_mirror dm_mod ext3 jbd aic7xxx scsi_transport_spi sd_mod scsi_mod
> May 17 11:49:42 malbec kernel: CPU: 1
> May 17 11:49:42 malbec kernel: EIP: 0060:[<c019797d>] Not tainted VLI
> May 17 11:49:42 malbec kernel: EFLAGS: 00010282 (2.6.16-1.2111_FC5smp #1)
> May 17 11:49:42 malbec kernel: EIP is at sysfs_readdir+0x153/0x202
> May 17 11:49:42 malbec kernel: eax: 00000000 ebx: d56cd5a0 ecx: 0000000a edx: 00000002
> May 17 11:49:42 malbec kernel: esi: e3a2e6fc edi: e3a2e707 ebp: e2c08b14 esp: d55a7f50
> May 17 11:49:42 malbec kernel: ds: 007b es: 007b ss: 0068
> May 17 11:49:42 malbec kernel: Process hald (pid: 2512, threadinfo=d55a7000 task=d8bc7930)
> May 17 11:49:42 malbec kernel: Stack: <0>c0171894 d55a7f98 d4c73e60 e3b28a4c e2c08b18 0000000a c0362f80 d4c73e60
> May 17 11:49:42 malbec kernel: e3b2655c e3b265dc c0171a53 d55a7f98 c0171894 fffffff7 09e4f5d4 00000000
> May 17 11:49:42 malbec kernel: d4c73e60 c0171ae0 09e4f754 09e4f73c 00000e80 ffffffea 0000000d 09e4f5d4
> May 17 11:49:42 malbec kernel: Call Trace:
> May 17 11:49:42 malbec kernel: [<c0171894>] filldir64+0x0/0xc3 [<c0171a53>] vfs_readdir+0x66/0x90
> May 17 11:49:42 malbec kernel: [<c0171894>] filldir64+0x0/0xc3 [<c0171ae0>] sys_getdents64+0x63/0xa5
> May 17 11:49:42 malbec kernel: [<c0103db9>] syscall_call+0x7/0xb <0>Code: 14 00 0f 84 ad 00 00 00 89 e8 e8 b3 e8 ff ff 89 c6 31 c0 83 c9 ff 89 f7 f2 ae f7 d1 49 89 4c 24 14 8b 45 20 85 c0 74 08 8b 40 18 <8b> 50 20 eb 11 ba 02 00 00 00 a1 20 24 48 c0 e8 2e 06 fe ff 89
A quick look at linux-2.6.19-rc2/fs/sysfs/dir.c::sysfs_readdir shows a
list_move, a traversal through a list, and some accesses to data with no
apparent protection by mutexes.
Is there reason to be worried...?
--
Stefan Richter
-=====-=-==- =-=- =-=-=
http://arcgraph.de/sr/
On Saturday 21 October 2006 10:43, Stefan Richter wrote:
> Here is a report about lockups and/or(?) oopses while HALD is obviously
> traversing sysfs files of FireWire devices, seen under Fedora Core
> kernels 2.6.{16,17,18} on an SMP setup:
>
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=188140
>
> > May 17 11:49:42 malbec kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000020
> > May 17 11:49:42 malbec kernel: printing eip:
> > May 17 11:49:42 malbec kernel: c019797d
> > May 17 11:49:42 malbec kernel: *pde = 14924001
> > May 17 11:49:42 malbec kernel: Oops: 0000 [#1]
> > May 17 11:49:42 malbec kernel: SMP
> > May 17 11:49:42 malbec kernel: last sysfs file: /class/net/eth0/address
> > May 17 11:49:42 malbec kernel: Modules linked in: sunrpc xt_limit xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables ip6table_filter ip6_tables x_tables ipv6 dm_multipath lp parport_pc parport floppy nvram ohci1394 ieee1394 uhci_hcd sg ohci_hcd ehci_hcd snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_emu10k1 snd_ac97_codec snd_ac97_bus snd_usb_audio snd_usb_lib snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event i2c_piix4 snd_seq i2c_core snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_device snd_timer snd_page_alloc snd_util_mem emu10k1_gp snd_hwdep gameport snd soundcore e100 mii dm_snapshot dm_zero dm_mirror dm_mod ext3 jbd aic7xxx scsi_transport_spi sd_mod scsi_mod
> > May 17 11:49:42 malbec kernel: CPU: 1
> > May 17 11:49:42 malbec kernel: EIP: 0060:[<c019797d>] Not tainted VLI
> > May 17 11:49:42 malbec kernel: EFLAGS: 00010282 (2.6.16-1.2111_FC5smp #1)
> > May 17 11:49:42 malbec kernel: EIP is at sysfs_readdir+0x153/0x202
> > May 17 11:49:42 malbec kernel: eax: 00000000 ebx: d56cd5a0 ecx: 0000000a edx: 00000002
> > May 17 11:49:42 malbec kernel: esi: e3a2e6fc edi: e3a2e707 ebp: e2c08b14 esp: d55a7f50
> > May 17 11:49:42 malbec kernel: ds: 007b es: 007b ss: 0068
> > May 17 11:49:42 malbec kernel: Process hald (pid: 2512, threadinfo=d55a7000 task=d8bc7930)
> > May 17 11:49:42 malbec kernel: Stack: <0>c0171894 d55a7f98 d4c73e60 e3b28a4c e2c08b18 0000000a c0362f80 d4c73e60
> > May 17 11:49:42 malbec kernel: e3b2655c e3b265dc c0171a53 d55a7f98 c0171894 fffffff7 09e4f5d4 00000000
> > May 17 11:49:42 malbec kernel: d4c73e60 c0171ae0 09e4f754 09e4f73c 00000e80 ffffffea 0000000d 09e4f5d4
> > May 17 11:49:42 malbec kernel: Call Trace:
> > May 17 11:49:42 malbec kernel: [<c0171894>] filldir64+0x0/0xc3 [<c0171a53>] vfs_readdir+0x66/0x90
> > May 17 11:49:42 malbec kernel: [<c0171894>] filldir64+0x0/0xc3 [<c0171ae0>] sys_getdents64+0x63/0xa5
> > May 17 11:49:42 malbec kernel: [<c0103db9>] syscall_call+0x7/0xb <0>Code: 14 00 0f 84 ad 00 00 00 89 e8 e8 b3 e8 ff ff 89 c6 31 c0 83 c9 ff 89 f7 f2 ae f7 d1 49 89 4c 24 14 8b 45 20 85 c0 74 08 8b 40 18 <8b> 50 20 eb 11 ba 02 00 00 00 a1 20 24 48 c0 e8 2e 06 fe ff 89
>
> A quick look at linux-2.6.19-rc2/fs/sysfs/dir.c::sysfs_readdir shows a
> list_move, a traversal through a list, and some accesses to data with no
> apparent protection by mutexes.
>
> Is there reason to be worried...?
Looking through the code I was not able to locate the exact line
at which it oopses, with the above oops message.
Are you able to track down the exact pointer dereference, which causes
this? By inserting printks, perhaps.
Maybe FC changed some of the structures. I couldn't find
a used structure with an interresting member at offset 00000020, at least.
--
Greetings Michael.
Michael Buesch wrote:
> On Saturday 21 October 2006 10:43, Stefan Richter wrote:
...
>> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=188140
...
> Looking through the code I was not able to locate the exact line
> at which it oopses, with the above oops message.
> Are you able to track down the exact pointer dereference, which causes
> this? By inserting printks, perhaps.
I need the original reporter to do this since I cannot reproduce the
bug. Probably because I don't have an SMP machine yet.
> Maybe FC changed some of the structures. I couldn't find
> a used structure with an interresting member at offset 00000020, at least.
Could be struct sysfs_dirent.s_dentry if I'm counting correctly in
http://www.linux-m32r.org/lxr/http/source/include/linux/sysfs.h?v=2.6.16#L68
The trace was from 2.6.16.
--
Stefan Richter
-=====-=-==- =-=- =-=-=
http://arcgraph.de/sr/
On Saturday 21 October 2006 23:09, Stefan Richter wrote:
> Michael Buesch wrote:
> > On Saturday 21 October 2006 10:43, Stefan Richter wrote:
> ...
> >> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=188140
> ...
> > Looking through the code I was not able to locate the exact line
> > at which it oopses, with the above oops message.
> > Are you able to track down the exact pointer dereference, which causes
> > this? By inserting printks, perhaps.
>
> I need the original reporter to do this since I cannot reproduce the
> bug. Probably because I don't have an SMP machine yet.
>
> > Maybe FC changed some of the structures. I couldn't find
> > a used structure with an interresting member at offset 00000020, at least.
>
> Could be struct sysfs_dirent.s_dentry if I'm counting correctly in
> http://www.linux-m32r.org/lxr/http/source/include/linux/sysfs.h?v=2.6.16#L68
> The trace was from 2.6.16.
Yeah, I found that offset, too, but:
There is only one usage of s_dentry
if (next->s_dentry)
But _before_ that there already comes
if (!next->s_element)
So, if "next" was NULL, it would already oops there.
--
Greetings Michael.
Michael Buesch wrote:
> On Saturday 21 October 2006 23:09, Stefan Richter wrote:
...
>>>> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=188140
[hard lockups, also one oops: 'Unable to handle kernel NULL pointer
dereference at virtual address 00000020', 'EIP is at sysfs_readdir',
'Process hald']
...
>>> Maybe FC changed some of the structures. I couldn't find
>>> a used structure with an interresting member at offset 00000020, at least.
>> Could be struct sysfs_dirent.s_dentry if I'm counting correctly in
>> http://www.linux-m32r.org/lxr/http/source/include/linux/sysfs.h?v=2.6.16#L68
>> The trace was from 2.6.16.
>
> Yeah, I found that offset, too, but:
>
> There is only one usage of s_dentry
> if (next->s_dentry)
>
> But _before_ that there already comes
> if (!next->s_element)
>
> So, if "next" was NULL, it would already oops there.
What if "next" became NULL afterwards? I know it's unlikely (but so is
the whole bug, given that we have just one reporter despite the bug's
age), but is it impossible? IOW does sysfs_readdir have any indirect
mutex protection?
Dave, do you patch sysfs datatypes in FC's kernel, or types they include?
--
Stefan Richter
-=====-=-==- =-=- =-==-
http://arcgraph.de/sr/
On Sunday 22 October 2006 11:08, Stefan Richter wrote:
> > Yeah, I found that offset, too, but:
> >
> > There is only one usage of s_dentry
> > if (next->s_dentry)
> >
> > But _before_ that there already comes
> > if (!next->s_element)
> >
> > So, if "next" was NULL, it would already oops there.
>
> What if "next" became NULL afterwards?
Hm, yeah. Makes kind of sense.
> I know it's unlikely (but so is
> the whole bug, given that we have just one reporter despite the bug's
> age), but is it impossible? IOW does sysfs_readdir have any indirect
> mutex protection?
I think it's protected by the BKL, but I dunno if that's sufficient here.
--
Greetings Michael.
On Sun, Oct 22, 2006 at 11:08:58AM +0200, Stefan Richter wrote:
> What if "next" became NULL afterwards? I know it's unlikely (but so is
> the whole bug, given that we have just one reporter despite the bug's
> age), but is it impossible? IOW does sysfs_readdir have any indirect
> mutex protection?
>
> Dave, do you patch sysfs datatypes in FC's kernel, or types they include?
Not that I recall. <linux/sysfs.h> is untouched in the trees I just
looked at (only have .18 ones handy right now)
Dave
--
http://www.codemonkey.org.uk
On Sat, Oct 21, 2006 at 10:43:49AM +0200, Stefan Richter wrote:
> Here is a report about lockups and/or(?) oopses while HALD is obviously
> traversing sysfs files of FireWire devices, seen under Fedora Core
> kernels 2.6.{16,17,18} on an SMP setup:
>
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=188140
>
> > May 17 11:49:42 malbec kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000020
> > May 17 11:49:42 malbec kernel: printing eip:
> > May 17 11:49:42 malbec kernel: c019797d
> > May 17 11:49:42 malbec kernel: *pde = 14924001
> > May 17 11:49:42 malbec kernel: Oops: 0000 [#1]
> > May 17 11:49:42 malbec kernel: SMP
> > May 17 11:49:42 malbec kernel: last sysfs file: /class/net/eth0/address
> > May 17 11:49:42 malbec kernel: Modules linked in: sunrpc xt_limit xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables ip6table_filter ip6_tables x_tables ipv6 dm_multipath lp parport_pc parport floppy nvram ohci1394 ieee1394 uhci_hcd sg ohci_hcd ehci_hcd snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul snd_emu10k1 snd_ac97_codec snd_ac97_bus snd_usb_audio snd_usb_lib snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event i2c_piix4 snd_seq i2c_core snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_device snd_timer snd_page_alloc snd_util_mem emu10k1_gp snd_hwdep gameport snd soundcore e100 mii dm_snapshot dm_zero dm_mirror dm_mod ext3 jbd aic7xxx scsi_transport_spi sd_mod scsi_mod
> > May 17 11:49:42 malbec kernel: CPU: 1
> > May 17 11:49:42 malbec kernel: EIP: 0060:[<c019797d>] Not tainted VLI
> > May 17 11:49:42 malbec kernel: EFLAGS: 00010282 (2.6.16-1.2111_FC5smp #1)
> > May 17 11:49:42 malbec kernel: EIP is at sysfs_readdir+0x153/0x202
> > May 17 11:49:42 malbec kernel: eax: 00000000 ebx: d56cd5a0 ecx: 0000000a edx: 00000002
> > May 17 11:49:42 malbec kernel: esi: e3a2e6fc edi: e3a2e707 ebp: e2c08b14 esp: d55a7f50
> > May 17 11:49:42 malbec kernel: ds: 007b es: 007b ss: 0068
> > May 17 11:49:42 malbec kernel: Process hald (pid: 2512, threadinfo=d55a7000 task=d8bc7930)
> > May 17 11:49:42 malbec kernel: Stack: <0>c0171894 d55a7f98 d4c73e60 e3b28a4c e2c08b18 0000000a c0362f80 d4c73e60
> > May 17 11:49:42 malbec kernel: e3b2655c e3b265dc c0171a53 d55a7f98 c0171894 fffffff7 09e4f5d4 00000000
> > May 17 11:49:42 malbec kernel: d4c73e60 c0171ae0 09e4f754 09e4f73c 00000e80 ffffffea 0000000d 09e4f5d4
> > May 17 11:49:42 malbec kernel: Call Trace:
> > May 17 11:49:42 malbec kernel: [<c0171894>] filldir64+0x0/0xc3 [<c0171a53>] vfs_readdir+0x66/0x90
> > May 17 11:49:42 malbec kernel: [<c0171894>] filldir64+0x0/0xc3 [<c0171ae0>] sys_getdents64+0x63/0xa5
> > May 17 11:49:42 malbec kernel: [<c0103db9>] syscall_call+0x7/0xb <0>Code: 14 00 0f 84 ad 00 00 00 89 e8 e8 b3 e8 ff ff 89 c6 31 c0 83 c9 ff 89 f7 f2 ae f7 d1 49 89 4c 24 14 8b 45 20 85 c0 74 08 8b 40 18 <8b> 50 20 eb 11 ba 02 00 00 00 a1 20 24 48 c0 e8 2e 06 fe ff 89
>
> A quick look at linux-2.6.19-rc2/fs/sysfs/dir.c::sysfs_readdir shows a
> list_move, a traversal through a list, and some accesses to data with no
> apparent protection by mutexes.
>
Most of the sysfs locking is done by parent inode's i_mutex. And for
sysfs_readdir(), i_mutex is held at VFS layer in vfs_readdir().
If this is not happening during boot, could you please recreate the
oops with at kdump? I guess kdump should work well on FC.
(http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28FC6KdumpKexecHowTo%29)
> Is there reason to be worried...?
Though locking looks fine to me.. but oops is a reason enough to
be worried about.
Thanks
Maneesh
Maneesh Soni wrote on 2006-10-30:
> On Sat, Oct 21, 2006 at 10:43:49AM +0200, Stefan Richter wrote:
...
>> A quick look at linux-2.6.19-rc2/fs/sysfs/dir.c::sysfs_readdir shows a
>> list_move, a traversal through a list, and some accesses to data with no
>> apparent protection by mutexes.
>>
> Most of the sysfs locking is done by parent inode's i_mutex. And for
> sysfs_readdir(), i_mutex is held at VFS layer in vfs_readdir().
OK, thanks.
> If this is not happening during boot, could you please recreate the
> oops with at kdump? I guess kdump should work well on FC.
The report came from someone else; I notified him. I cannot reproduce
it, I don't have FC nor hald. BTW the reporter mentioned that the
machine freezes completely much more often than merely throwing an oops.
(Both caused by hald.)
> (http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28FC6KdumpKexecHowTo%29)
>
>> Is there reason to be worried...?
>
> Though locking looks fine to me.. but oops is a reason enough to
> be worried about.
I referred to the locking. :-) Thanks for the hints,
--
Stefan Richter
-=====-=-==- =-=- =====
http://arcgraph.de/sr/