2008-10-17 19:20:42

by Alexander Beregalov

[permalink] [raw]
Subject: 2.6.27-05178-g2e532d6: list_add corruption

I am afraid I do not know how to reproduce this.

WARNING: at lib/list_debug.c:30 __list_add+0x44/0x5c()
list_add corruption. prev->next should be next (f6638448), but was
f26f078c. (prev=f26f078c).
Modules linked in: w83627hf hwmon_vid i2c_nforce2
Pid: 1933, comm: rtorrent Not tainted 2.6.27-05178-g2e532d6 #6
Call Trace:
[<c011de4b>] warn_slowpath+0x4b/0x69
[<c0137a03>] ? trace_hardirqs_off+0xb/0xd
[<c01070ab>] ? native_sched_clock+0x76/0x88
[<c0138cc5>] ? trace_hardirqs_on+0xb/0xd
[<c0138c93>] ? trace_hardirqs_on_caller+0xe6/0x10d
[<c031aa43>] ? release_sock+0xac/0xb4
[<c0138cc5>] ? trace_hardirqs_on+0xb/0xd
[<c0139d4e>] ? __lock_acquire+0xaeb/0xafa
[<c0139d4e>] ? __lock_acquire+0xaeb/0xafa
[<c0137a39>] ? put_lock_stats+0xd/0x21
[<c026a404>] __list_add+0x44/0x5c
[<c0186be0>] ep_poll_callback+0x5b/0xb4
[<c011735b>] __wake_up_common+0x35/0x5b
[<c0117c6d>] __wake_up_sync+0x31/0x44
[<c031ae39>] sock_def_readable+0x38/0x63
[<c03467eb>] tcp_rcv_established+0x37c/0x49d
[<c034c2cc>] tcp_v4_do_rcv+0x25/0x151
[<c0386269>] ? _spin_lock_nested+0x51/0x59
[<c034c6e9>] tcp_v4_rcv+0x2f1/0x4a2
[<c0336a65>] ip_local_deliver+0x86/0xd6
[<c03369c1>] ip_rcv+0x361/0x37f
[<c0322ad7>] netif_receive_skb+0x1af/0x1cf
[<c02e5892>] nv_napi_poll+0x201/0x4f8
[<c0138cc5>] ? trace_hardirqs_on+0xb/0xd
[<c03217f0>] net_rx_action+0x98/0x155
[<c0121d94>] __do_softirq+0x6a/0xf1
[<c0121e4a>] do_softirq+0x2f/0x47
[<c0121fb9>] irq_exit+0x3b/0x84
[<c01046fb>] do_IRQ+0x6d/0x82
[<c01038d0>] common_interrupt+0x28/0x30
---[ end trace 71f919bde0b515ac ]---
------------[ cut here ]------------
WARNING: at lib/list_debug.c:30 __list_add+0x44/0x5c()
list_add corruption. prev->next should be next (f6638448), but was
f26f078c. (prev=f26f078c).
Modules linked in: w83627hf hwmon_vid i2c_nforce2
Pid: 1933, comm: rtorrent Tainted: G W 2.6.27-05178-g2e532d6 #6
Call Trace:
[<c011de4b>] warn_slowpath+0x4b/0x69
[<c0160063>] ? shmem_writepage+0x147/0x1ad
[<c0139de8>] ? print_lock_contention_bug+0x14/0xd7
[<c0138cc5>] ? trace_hardirqs_on+0xb/0xd
[<c0137a39>] ? put_lock_stats+0xd/0x21
[<c026a404>] __list_add+0x44/0x5c
[<c0187600>] sys_epoll_ctl+0x36f/0x405
[<c0137a03>] ? trace_hardirqs_off+0xb/0xd
[<c0102ea9>] sysenter_do_call+0x12/0x35
---[ end trace 71f919bde0b515ac ]---
------------[ cut here ]------------
WARNING: at lib/list_debug.c:30 __list_add+0x44/0x5c()
list_add corruption. prev->next should be next (f6638448), but was
f26e4a8c. (prev=f26e4a8c).
Modules linked in: w83627hf hwmon_vid i2c_nforce2
Pid: 1933, comm: rtorrent Tainted: G W 2.6.27-05178-g2e532d6 #6
Call Trace:
[<c011de4b>] warn_slowpath+0x4b/0x69
[<c0138cc5>] ? trace_hardirqs_on+0xb/0xd
[<c0321890>] ? net_rx_action+0x138/0x155
[<c0121d28>] ? _local_bh_enable+0x86/0x88
[<c0137a39>] ? put_lock_stats+0xd/0x21
[<c026a404>] __list_add+0x44/0x5c
[<c0187600>] sys_epoll_ctl+0x36f/0x405
[<c0102ea9>] sysenter_do_call+0x12/0x35
---[ end trace 71f919bde0b515ac ]---
------------[ cut here ]------------
WARNING: at lib/list_debug.c:30 __list_add+0x44/0x5c()
list_add corruption. prev->next should be next (f6638448), but was
f26e4a8c. (prev=f26e4a8c).
Modules linked in: w83627hf hwmon_vid i2c_nforce2
Pid: 1933, comm: rtorrent Tainted: G W 2.6.27-05178-g2e532d6 #6
Call Trace:
[<c011de4b>] warn_slowpath+0x4b/0x69
[<c0138cc5>] ? trace_hardirqs_on+0xb/0xd
[<c0321890>] ? net_rx_action+0x138/0x155
[<c0121d28>] ? _local_bh_enable+0x86/0x88
[<c0137a39>] ? put_lock_stats+0xd/0x21
[<c026a404>] __list_add+0x44/0x5c
[<c0187600>] sys_epoll_ctl+0x36f/0x405
[<c0102ea9>] sysenter_do_call+0x12/0x35
---[ end trace 71f919bde0b515ac ]---
BUG: unable to handle kernel paging request at 6b6b6b7b
IP: [<c0187068>] sys_epoll_wait+0x24c/0x3c2
*pde = 00000000
Oops: 0000 [#1] PREEMPT DEBUG_PAGEALLOC
last sysfs file: /sys/devices/platform/w83627hf.656/name
Modules linked in: w83627hf hwmon_vid i2c_nforce2

Pid: 1933, comm: rtorrent Tainted: G W (2.6.27-05178-g2e532d6 #6)
EIP: 0060:[<c0187068>] EFLAGS: 00010246 CPU: 0
EIP is at sys_epoll_wait+0x24c/0x3c2
EAX: 6b6b6b6b EBX: f26e4a8c ECX: f673e000 EDX: 00000000
ESI: f26e4a80 EDI: f6638390 EBP: f673ffb0 ESP: f673ff4c
DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
Process rtorrent (pid: 1933, ti=f673e000 task=f6735100 task.ti=f673e000)
Stack:
f673ff9c c0176bda 00000000 f670ef00 f6638418 0000000b f6638448 f6638440
f26e4a8c f6638420 f66383f8 00000008 f66383b0 f673ff94 c025db74 f673ff9c
bfbed370 00000000 f673ffb0 bfbed370 f673ff9c f673ff9c 00000006 00000022
Call Trace:
[<c0176bda>] ? mntput_no_expire+0x16/0xb4
[<c025db74>] ? copy_to_user+0x36/0x106
[<c0102ea9>] ? sysenter_do_call+0x12/0x35
Code: c7 45 a4 00 00 00 00 e9 8e 00 00 00 8b 75 bc 83 ee 0c 8b 56 0c
8b 46 10 89 42 04 89 10 8b 55 bc 8b 46 18 89 56 0c 89 56 10 31 d2 <8b>
48 10 ff 51 1c 8b 5e 38 21 d8 89 5d c8 89 45 d0 74 3f 6b 45
EIP: [<c0187068>] sys_epoll_wait+0x24c/0x3c2 SS:ESP 0068:f673ff4c


2008-10-17 23:19:23

by Davide Libenzi

[permalink] [raw]
Subject: Re: 2.6.27-05178-g2e532d6: list_add corruption

On Fri, 17 Oct 2008, Alexander Beregalov wrote:

> I am afraid I do not know how to reproduce this.
>
> WARNING: at lib/list_debug.c:30 __list_add+0x44/0x5c()
> list_add corruption. prev->next should be next (f6638448), but was
> f26f078c. (prev=f26f078c).
> Modules linked in: w83627hf hwmon_vid i2c_nforce2
> Pid: 1933, comm: rtorrent Not tainted 2.6.27-05178-g2e532d6 #6
> Call Trace:
> [<c011de4b>] warn_slowpath+0x4b/0x69
> [<c0137a03>] ? trace_hardirqs_off+0xb/0xd
> [<c01070ab>] ? native_sched_clock+0x76/0x88
> [<c0138cc5>] ? trace_hardirqs_on+0xb/0xd
> [<c0138c93>] ? trace_hardirqs_on_caller+0xe6/0x10d
> [<c031aa43>] ? release_sock+0xac/0xb4
> [<c0138cc5>] ? trace_hardirqs_on+0xb/0xd
> [<c0139d4e>] ? __lock_acquire+0xaeb/0xafa
> [<c0139d4e>] ? __lock_acquire+0xaeb/0xafa
> [<c0137a39>] ? put_lock_stats+0xd/0x21
> [<c026a404>] __list_add+0x44/0x5c
> [<c0186be0>] ep_poll_callback+0x5b/0xb4

This is very likely to be caused by a patch that went in a few days ago.
Already posted a fix to Andrew.



- Davide

2008-10-18 06:41:23

by Alexander Beregalov

[permalink] [raw]
Subject: Re: 2.6.27-05178-g2e532d6: list_add corruption

> This is very likely to be caused by a patch that went in a few days ago.> Already posted a fix to Andrew.>
Is it the same problem? 2.6.27-05577-g0cfd810
BUG eventpoll_epi: Poison overwritten-----------------------------------------------------------------------------INFO: 0xf2ac768c-0xf2ac7693. First byte 0xc instead of 0x6bINFO: Allocated in sys_epoll_ctl+0x120/0x405 age=21753 cpu=0 pid=1936INFO: Freed in ep_remove+0x7f/0x86 age=3964 cpu=0 pid=1936INFO: Slab 0xc16568e0 objects=32 used=29 fp=0xf2ac7680 flags=0x400000c3INFO: Object 0xf2ac7680 @offset=1664 fp=0xf2ac7f80
Bytes b4 0xf2ac7670: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5aZZZZZZZZZZZZZZZZ Object 0xf2ac7680: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 0c 7d b1 f2kkkkkkkkkkkk.}╠Р Object 0xf2ac7690: 0c 15 a7 f2 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b..╖Рkkkkkkkkkkkk Object 0xf2ac76a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6bkkkkkkkkkkkkkkkk Object 0xf2ac76b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6bkkkkkkkkkkkkkkkk Object 0xf2ac76c0: 6b 6b 6b a5 kkk╔ Redzone 0xf2ac76c4: bb bb bb bb ╩╩╩╩ Padding 0xf2ac76ec: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5aZZZZZZZZZZZZZZZZ Padding 0xf2ac76fc: 5a 5a 5a 5a ZZZZPid: 1936, comm: rtorrent Not tainted 2.6.27-05577-g0cfd810 #1Call Trace: [<c0161628>] print_trailer+0xe1/0xe9 [<c01616b1>] check_bytes_and_report+0x81/0xa4 [<c01617a9>] check_object+0xa5/0x160 [<c0162ceb>] __slab_alloc+0x420/0x4c7 [<c0138d45>] ? trace_hardirqs_on+0xb/0xd [<c01630d4>] kmem_cache_alloc+0x62/0xaf [<c0187485>] ? sys_epoll_ctl+0x120/0x405 [<c0187485>] ? sys_epoll_ctl+0x120/0x405 [<c0187485>] sys_epoll_ctl+0x120/0x405 [<c025d720>] ? trace_hardirqs_on_thunk+0xc/0x10 [<c0102ea9>] sysenter_do_call+0x12/0x35FIX eventpoll_epi: Restoring 0xf2ac768c-0xf2ac7693=0x6b
FIX eventpoll_epi: Marking all objects used????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay?ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m???? ????????I?

2008-10-18 14:38:59

by Davide Libenzi

[permalink] [raw]
Subject: Re: 2.6.27-05178-g2e532d6: list_add corruption

On Sat, 18 Oct 2008, Alexander Beregalov wrote:

> > This is very likely to be caused by a patch that went in a few days ago.
> > Already posted a fix to Andrew.
> >
>
> Is it the same problem? 2.6.27-05577-g0cfd810

Hard to be sure, but certainly a double insert can create a similar
problem. The only way to tell is to get the fix and retry.


- Davide

2008-10-18 15:50:32

by Alexander Beregalov

[permalink] [raw]
Subject: Re: 2.6.27-05178-g2e532d6: list_add corruption

2008/10/18 Davide Libenzi <[email protected]>:
> On Sat, 18 Oct 2008, Alexander Beregalov wrote:
>
>> > This is very likely to be caused by a patch that went in a few days ago.
>> > Already posted a fix to Andrew.
>> >
>>
>> Is it the same problem? 2.6.27-05577-g0cfd810
>
> Hard to be sure, but certainly a double insert can create a similar
> problem. The only way to tell is to get the fix and retry.
I have already applied your patch and the kernel works better, no
errors anymore.
Thanks