2005-01-25 04:45:22

by Zwane Mwaikambo

[permalink] [raw]
Subject: 2.6.11-rc2-mm1 kernel BUG at kernel/workqueue.c:104

I pressed a key on a VT during boot and got the following;

usb-storage: device scan complete
------------[ cut here ]------------
kernel BUG at kernel/workqueue.c:104!
invalid operand: 0000 [#1]
PREEMPT SMP DEBUG_PAGEALLOC
Modules linked in:
CPU: 0
EIP: 0060:[<c01314af>] Not tainted VLI
EFLAGS: 00010017 (2.6.11-rc2-mm1)
EIP is at queue_work+0x8f/0xa0
eax: f5ebe0e4 ebx: f5ebe0e0 ecx: 00000001 edx: 00000000
esi: c19adef8 edi: 00000000 ebp: f5e4fe14 esp: f5e4fe08
ds: 007b es: 007b ss: 0068
Process udev.hotplug (pid: 1351, threadinfo=f5e4e000 task=f5817ac0)
Stack: c1815000 0000001c 00000001 f5e4fe20 c0388d3d c1815000 f5e4fe2c c03890c1
00000002 f5e4fe64 c0389be1 f5e4ffc4 c011a7b8 00000000 00000002 00000086
00000001 00000020 c1815000 f5e4ffc4 00000001 f7b09674 0000001c f5e4fe80
Call Trace:
[<c010403a>] show_stack+0x7a/0x90
[<c01041c6>] show_registers+0x156/0x1c0
[<c01043e0>] die+0x100/0x190
[<c0104819>] do_invalid_op+0xa9/0xc0
[<c0103cc7>] error_code+0x2b/0x30
[<c0388d3d>] fn_enter+0x1d/0x60
[<c03890c1>] k_spec+0x31/0x50
[<c0389be1>] kbd_keycode+0x1a1/0x300
[<c0389dd3>] kbd_event+0x93/0xf0
[<c04d2963>] input_event+0xd3/0x3b0
[<c04d74cf>] atkbd_report_key+0x2f/0x70
[<c04d76fe>] atkbd_interrupt+0x1ee/0x510
[<c03abbf3>] serio_interrupt+0x43/0x74
[<c03ac84d>] i8042_interrupt+0x17d/0x270
[<c0141e8a>] handle_IRQ_event+0x2a/0x60
[<c0141fa9>] __do_IRQ+0xe9/0x140
[<c0105513>] do_IRQ+0x33/0x70
[<c0103b92>] common_interrupt+0x1a/0x20
Code: e8 37 92 fe ff b8 00 e0 ff ff 21 e0 8b 40 08 a8 08 75 11 89 f8 8b 1c
24 8b 74 24 04 8b 7c 2408 89 ec 5d c3 e8 33 7b 4b 00 eb e8 <0f> 0b 68 00
a3 55 62 c0 eb b1 8d b4 26 00 00 00 00 55 89 e5 83
<0>Kernel panic - not syncing: Fatal exception in interrupt


2005-01-25 05:44:33

by Andrew Morton

[permalink] [raw]
Subject: Re: 2.6.11-rc2-mm1 kernel BUG at kernel/workqueue.c:104

Zwane Mwaikambo <[email protected]> wrote:
>
> I pressed a key on a VT during boot and got the following;
>
> usb-storage: device scan complete
> ------------[ cut here ]------------
> kernel BUG at kernel/workqueue.c:104!
> invalid operand: 0000 [#1]
> PREEMPT SMP DEBUG_PAGEALLOC
> Modules linked in:
> CPU: 0
> EIP: 0060:[<c01314af>] Not tainted VLI
> EFLAGS: 00010017 (2.6.11-rc2-mm1)
> EIP is at queue_work+0x8f/0xa0
> eax: f5ebe0e4 ebx: f5ebe0e0 ecx: 00000001 edx: 00000000
> esi: c19adef8 edi: 00000000 ebp: f5e4fe14 esp: f5e4fe08
> ds: 007b es: 007b ss: 0068
> Process udev.hotplug (pid: 1351, threadinfo=f5e4e000 task=f5817ac0)
> Stack: c1815000 0000001c 00000001 f5e4fe20 c0388d3d c1815000 f5e4fe2c c03890c1
> 00000002 f5e4fe64 c0389be1 f5e4ffc4 c011a7b8 00000000 00000002 00000086
> 00000001 00000020 c1815000 f5e4ffc4 00000001 f7b09674 0000001c f5e4fe80
> Call Trace:
> [<c010403a>] show_stack+0x7a/0x90
> [<c01041c6>] show_registers+0x156/0x1c0
> [<c01043e0>] die+0x100/0x190
> [<c0104819>] do_invalid_op+0xa9/0xc0
> [<c0103cc7>] error_code+0x2b/0x30
> [<c0388d3d>] fn_enter+0x1d/0x60
> [<c03890c1>] k_spec+0x31/0x50
> [<c0389be1>] kbd_keycode+0x1a1/0x300
> [<c0389dd3>] kbd_event+0x93/0xf0
> [<c04d2963>] input_event+0xd3/0x3b0
> [<c04d74cf>] atkbd_report_key+0x2f/0x70
> [<c04d76fe>] atkbd_interrupt+0x1ee/0x510
> [<c03abbf3>] serio_interrupt+0x43/0x74
> [<c03ac84d>] i8042_interrupt+0x17d/0x270
> [<c0141e8a>] handle_IRQ_event+0x2a/0x60

I can't reproduce it from a quick test here. I'd assume that the keystroke
came in before the vt's workqueue is initialised. fn_enter() calls
put_queue() calls con_schedule_flip() calls schedule_work() which goes BUG:

BUG_ON(!list_empty(&work->entry));

I bet the list_head pointers are null.

Roman might have a theory?

2005-01-25 07:23:43

by Zwane Mwaikambo

[permalink] [raw]
Subject: Re: 2.6.11-rc2-mm1 kernel BUG at kernel/workqueue.c:104

On Mon, 24 Jan 2005, Andrew Morton wrote:

> I can't reproduce it from a quick test here. I'd assume that the keystroke
> came in before the vt's workqueue is initialised. fn_enter() calls
> put_queue() calls con_schedule_flip() calls schedule_work() which goes BUG:

Boot into runlevel 1 (console will then be on serial, nothing on any of
the VTs), then press a key. This can be any time after it's booted into
runlevel 1.

Thanks,
Zwane

2005-01-25 08:15:03

by Andrew Morton

[permalink] [raw]
Subject: Re: 2.6.11-rc2-mm1 kernel BUG at kernel/workqueue.c:104

Zwane Mwaikambo <[email protected]> wrote:
>
> On Mon, 24 Jan 2005, Andrew Morton wrote:
>
> > I can't reproduce it from a quick test here. I'd assume that the keystroke
> > came in before the vt's workqueue is initialised. fn_enter() calls
> > put_queue() calls con_schedule_flip() calls schedule_work() which goes BUG:
>
> Boot into runlevel 1 (console will then be on serial, nothing on any of
> the VTs), then press a key. This can be any time after it's booted into
> runlevel 1.
>

OK, thanks. I get what appears to be a use-after-free error.
CONFIG_DEBUG_PAGEALLOC is set:

Program received signal SIGEMT, Emulation trap.
0xc0272bc2 in kbd_keycode (keycode=57, down=1, hw_raw=0, regs=0xc0673f9c)
at drivers/char/keyboard.c:1035
1035 if (tty && (!tty->driver_data)) {
(gdb) p tty
$1 = (struct tty_struct *) 0xce3c4000
(gdb) p *tty
Cannot access memory at address 0xce3c4000
(gdb) bt
#0 0xc0272bc2 in kbd_keycode (keycode=57, down=1, hw_raw=0, regs=0xc0673f9c)
at drivers/char/keyboard.c:1035
#1 0xc0272ee4 in kbd_event (handle=0xcf150674, event_type=1, event_code=57,
value=1) at drivers/char/keyboard.c:1162
#2 0xc03081d8 in input_event (dev=0xcf19b090, type=1, code=57, value=1)
at drivers/input/input.c:188
#3 0xc030a71a in atkbd_report_key (dev=0xcf19b090, regs=0xc1235000, code=57,
value=0) at drivers/input/keyboard/atkbd.c:239
#4 0xc030ab8b in atkbd_interrupt (serio=0xcf771df8, data=57 '9', flags=0,
regs=0xc0673f9c) at drivers/input/keyboard/atkbd.c:392
#5 0xc0279dd9 in serio_interrupt (serio=0xcf771df8, data=57 '9', dfl=0,
regs=0xc1235000) at drivers/input/serio/serio.c:681
#6 0xc027a96f in i8042_interrupt (irq=1, dev_id=0xc06cb3a0, regs=0xc1235000)
at drivers/input/serio/i8042.c:481
#7 0xc013b7e5 in handle_IRQ_event (irq=1, regs=0xc0673f9c, action=0xcf0ee85c)
at kernel/irq/handle.c:90
#8 0xc013b913 in __do_IRQ (irq=1, regs=0xc0673f9c) at kernel/irq/handle.c:177
#9 0xc0104eee in do_IRQ (regs=0x0) at arch/i386/kernel/irq.c:105
#10 0xc010375a in common_interrupt () at arch/i386/kernel/semaphore.c:177

Roman, binary searching indicates that the bug was introduced by
merge-vt_struct-into-vc_data.patch. The latest version.

2005-01-25 10:29:57

by Roman Zippel

[permalink] [raw]
Subject: Re: 2.6.11-rc2-mm1 kernel BUG at kernel/workqueue.c:104

Hi,

On Tue, 25 Jan 2005, Andrew Morton wrote:

> Roman, binary searching indicates that the bug was introduced by
> merge-vt_struct-into-vc_data.patch. The latest version.

Ok, that bug is now really embarrassing. :)
I removed a bit too much from con_close.

bye, Roman

diff -ur -X /home/devel/roman/nodiff linux-2.6.11-rc2-mm1.org/drivers/char/vt.c linux-2.6.11-rc2-mm1/drivers/char/vt.c
--- linux-2.6.11-rc2-mm1.org/drivers/char/vt.c 2005-01-24 14:16:18.000000000 +0100
+++ linux-2.6.11-rc2-mm1/drivers/char/vt.c 2005-01-25 11:10:22.000000000 +0100
@@ -2474,6 +2476,10 @@
down(&tty_sem);
acquire_console_sem();
if (tty && tty->count == 1) {
+ struct vc_data *vc = tty->driver_data;
+
+ if (vc)
+ vc->vc_tty = NULL;
tty->driver_data = NULL;
release_console_sem();
vcs_remove_devfs(tty);


Attachments:
merge-vt_struct-into-vc_data.patch (27.37 kB)

2005-01-25 14:36:35

by Zwane Mwaikambo

[permalink] [raw]
Subject: Re: 2.6.11-rc2-mm1 kernel BUG at kernel/workqueue.c:104

On Tue, 25 Jan 2005, Andrew Morton wrote:

> OK, thanks. I get what appears to be a use-after-free error.
> CONFIG_DEBUG_PAGEALLOC is set:
>
> Program received signal SIGEMT, Emulation trap.
> 0xc0272bc2 in kbd_keycode (keycode=57, down=1, hw_raw=0, regs=0xc0673f9c)
> at drivers/char/keyboard.c:1035
> 1035 if (tty && (!tty->driver_data)) {
> (gdb) p tty
> $1 = (struct tty_struct *) 0xce3c4000
> (gdb) p *tty
> Cannot access memory at address 0xce3c4000
> (gdb) bt
> #0 0xc0272bc2 in kbd_keycode (keycode=57, down=1, hw_raw=0, regs=0xc0673f9c)
> at drivers/char/keyboard.c:1035
> #1 0xc0272ee4 in kbd_event (handle=0xcf150674, event_type=1, event_code=57,
> value=1) at drivers/char/keyboard.c:1162
> #2 0xc03081d8 in input_event (dev=0xcf19b090, type=1, code=57, value=1)
> at drivers/input/input.c:188
> #3 0xc030a71a in atkbd_report_key (dev=0xcf19b090, regs=0xc1235000, code=57,
> value=0) at drivers/input/keyboard/atkbd.c:239
> #4 0xc030ab8b in atkbd_interrupt (serio=0xcf771df8, data=57 '9', flags=0,
> regs=0xc0673f9c) at drivers/input/keyboard/atkbd.c:392
> #5 0xc0279dd9 in serio_interrupt (serio=0xcf771df8, data=57 '9', dfl=0,
> regs=0xc1235000) at drivers/input/serio/serio.c:681
> #6 0xc027a96f in i8042_interrupt (irq=1, dev_id=0xc06cb3a0, regs=0xc1235000)
> at drivers/input/serio/i8042.c:481
> #7 0xc013b7e5 in handle_IRQ_event (irq=1, regs=0xc0673f9c, action=0xcf0ee85c)
> at kernel/irq/handle.c:90
> #8 0xc013b913 in __do_IRQ (irq=1, regs=0xc0673f9c) at kernel/irq/handle.c:177
> #9 0xc0104eee in do_IRQ (regs=0x0) at arch/i386/kernel/irq.c:105
> #10 0xc010375a in common_interrupt () at arch/i386/kernel/semaphore.c:177
>
> Roman, binary searching indicates that the bug was introduced by
> merge-vt_struct-into-vc_data.patch. The latest version.

Thanks for narrowing it down Andrew.