Here is an Oops from the kernel 2.6.0-test9 + cset-20031115_0206.txt.gz (means all current patches till 03/11/14 [PATCH] PPC32: cancel syscall restart on signal delivery ).
Seems to come from the cache_flusharray function of mm/slab.c
It occurs in general during heavy load (compilation of kernel or xfree86...), is quite reproductible but not automatic.
Same Oops on different distro/gcc (slack 9.1 et lfs 5.0).
Arch is i386 (AMD athlon-tbird 1.2GHz)
I never had this problem with 2.6.0-test8 and before.
This oops is very annoying, I got it around 5 times a day. I'm sorry but I don't have the skill to patch this. If someone can help :-) Please CC me in answer, I'm not on the lkml.
Regards,
Jerome Pinot
---
ksymoops 2.4.9 on i686 2.6.0-test9. Options used
-V (default)
-K (specified)
-l /proc/modules (default)
-o /mnt/lfs/lib/modules/2.6.0-test9 (specified)
-m /mnt/lfs/boot/System.map (specified)
No modules in ksyms, skipping objects
No ksyms, skipping lsmod
kernel BUG at mm/slab.c:1957!
invalid operand: 0000 [#1]
CPU: 0
EIP: 0060:[free_block+336/752] Not tainted
EIP: 0060:[<c015ad40>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010096
eax: 00000045 ebx: 00000006 ecx: c0693854 edx: c056e4f8
esi: cd09a000 edi: cd09a018 ebp: cf821c68 esp: cf821c3c
ds: 007b es: 007b ss: 0068
Stack: c0502240 c0502e1d cd09af18 c0652a00 00000001 0000003a cd09af18 0000000f
cffdef08 c4bcd180 00000010 cf821ca0 c015afba cffed800 cffdef08 00000010
00000282 c1161ca0 00000000 00000001 cffee730 00000010 00010c00 c4bcd180
Call Trace:
[<c015afba>] cache_flusharray+0xda/0x2b0
[<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
[<c018158c>] free_buffer_head+0x2c/0x60
[<c018158c>] free_buffer_head+0x2c/0x60
[<c0181353>] try_to_free_buffers+0x143/0x220
[<c0294bab>] linvfs_release_page+0x7b/0x80
[<c017f243>] try_to_release_page+0x53/0x60
[<c015fec7>] shrink_list+0x8b7/0xac0
[<c010cadc>] common_interrupt+0x18/0x20
[<c015e489>] __pagevec_release+0x29/0x40
[<c01602d5>] shrink_cache+0x205/0x610
[<c0161228>] shrink_zone+0x78/0xa0
[<c01615fa>] balance_pgdat+0x17a/0x200
[<c0161759>] kswapd+0xd9/0xf0
[<c01274f0>] autoremove_wake_function+0x0/0x50
[<c010c046>] ret_from_fork+0x6/0x14
[<c01274f0>] autoremove_wake_function+0x0/0x50
[<c0161680>] kswapd+0x0/0xf0
[<c01092a9>] kernel_thread_helper+0x5/0xc
Code: 0f 0b a5 07 82 15 50 c0 8b 46 14 8b 4d e8 31 db 89 04 8f 89
>>EIP; c015ad40 <free_block+150/2f0> <=====
>>ecx; c0693854 <per_cpu__runqueues+4b4/940>
>>edx; c056e4f8 <log_wait+18/20>
>>esi; cd09a000 <__crc_unregister_sound_dsp+164f0/974e1>
>>edi; cd09a018 <__crc_unregister_sound_dsp+16508/974e1>
>>ebp; cf821c68 <__crc_scsi_register_driver+6c7b9/1f2ec0>
>>esp; cf821c3c <__crc_scsi_register_driver+6c78d/1f2ec0>
Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c010cadc <common_interrupt+18/20>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01615fa <balance_pgdat+17a/200>
Trace; c0161759 <kswapd+d9/f0>
Trace; c01274f0 <autoremove_wake_function+0/50>
Trace; c010c046 <ret_from_fork+6/14>
Trace; c01274f0 <autoremove_wake_function+0/50>
Trace; c0161680 <kswapd+0/f0>
Trace; c01092a9 <kernel_thread_helper+5/c>
Code; c015ad40 <free_block+150/2f0>
00000000 <_EIP>:
Code; c015ad40 <free_block+150/2f0> <=====
0: 0f 0b ud2a <=====
Code; c015ad42 <free_block+152/2f0>
2: a5 movsl %ds:(%esi),%es:(%edi)
Code; c015ad43 <free_block+153/2f0>
3: 07 pop %es
Code; c015ad44 <free_block+154/2f0>
4: 82 (bad)
Code; c015ad45 <free_block+155/2f0>
5: 15 50 c0 8b 46 adc $0x468bc050,%eax
Code; c015ad4a <free_block+15a/2f0>
a: 14 8b adc $0x8b,%al
Code; c015ad4c <free_block+15c/2f0>
c: 4d dec %ebp
Code; c015ad4d <free_block+15d/2f0>
d: e8 31 db 89 04 call 489db43 <_EIP+0x489db43>
Code; c015ad52 <free_block+162/2f0>
12: 8f 89 00 00 00 00 popl 0x0(%ecx)
Unable to handle kernel paging request at virtual address 00100104
c015ac3c
*pde = 00000000
Oops: 0002 [#2]
CPU: 0
EIP: 0060:[free_block+76/752] Not tainted
EIP: 0060:[<c015ac3c>] Not tainted
EFLAGS: 00010017
eax: 00100100 ebx: 00000000 ecx: cd09a810 edx: 00200200
esi: cd09a000 edi: c7430018 ebp: c558bafc esp: c558bad0
ds: 007b es: 007b ss: 0068
Stack: c558badc 00000086 80010c00 ccf77000 cffdc080 0000001a cd09a810 0000000b
cffdef08 cd09a180 00000010 c558bb34 c015afba cffed800 cffdef08 00000010
00000286 c11d0c68 00000000 00000001 cffee730 00000010 00010c00 cd09a180
Call Trace:
[<c015afba>] cache_flusharray+0xda/0x2b0
[<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
[<c018158c>] free_buffer_head+0x2c/0x60
[<c018158c>] free_buffer_head+0x2c/0x60
[<c0181353>] try_to_free_buffers+0x143/0x220
[<c0294bab>] linvfs_release_page+0x7b/0x80
[<c017f243>] try_to_release_page+0x53/0x60
[<c015fec7>] shrink_list+0x8b7/0xac0
[<c0123eef>] schedule+0x36f/0x8a0
[<c015e489>] __pagevec_release+0x29/0x40
[<c01602d5>] shrink_cache+0x205/0x610
[<c0127515>] autoremove_wake_function+0x25/0x50
[<c0161228>] shrink_zone+0x78/0xa0
[<c01612ee>] shrink_caches+0x9e/0xc0
[<c01613b7>] try_to_free_pages+0xa7/0x170
[<c015541a>] __alloc_pages+0x1fa/0x380
[<c0165c17>] do_anonymous_page+0xc7/0x520
[<c017a899>] do_sync_write+0x89/0xc0
[<c01668f2>] handle_mm_fault+0x132/0x310
[<c0121940>] do_page_fault+0x350/0x56a
[<c01698de>] do_brk+0x14e/0x230
[<c016790e>] sys_brk+0xee/0x120
[<c01215f0>] do_page_fault+0x0/0x56a
[<c010cb79>] error_code+0x2d/0x38
Code: 89 50 04 89 02 c7 46 04 00 02 20 00 c7 06 00 01 10 00 89 c8
>>EIP; c015ac3c <free_block+4c/2f0> <=====
>>eax; 00100100 <__crc_prepare_to_wait_exclusive+ce3e5/1424fd>
>>ecx; cd09a810 <__crc_unregister_sound_dsp+16d00/974e1>
>>edx; 00200200 <__crc___user_walk+3d8ad/1cb584>
>>esi; cd09a000 <__crc_unregister_sound_dsp+164f0/974e1>
>>edi; c7430018 <__crc_inet_getsockopt+60fc6/20c3c2>
>>ebp; c558bafc <__crc_fat_write_inode+336fc/668510>
>>esp; c558bad0 <__crc_fat_write_inode+336d0/668510>
Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Code; c015ac3c <free_block+4c/2f0>
00000000 <_EIP>:
Code; c015ac3c <free_block+4c/2f0> <=====
0: 89 50 04 mov %edx,0x4(%eax) <=====
Code; c015ac3f <free_block+4f/2f0>
3: 89 02 mov %eax,(%edx)
Code; c015ac41 <free_block+51/2f0>
5: c7 46 04 00 02 20 00 movl $0x200200,0x4(%esi)
Code; c015ac48 <free_block+58/2f0>
c: c7 06 00 01 10 00 movl $0x100100,(%esi)
Code; c015ac4e <free_block+5e/2f0>
12: 89 c8 mov %ecx,%eax
Call Trace:
[<c0124411>] schedule+0x891/0x8a0
[<c0163ae1>] unmap_page_range+0x41/0x70
[<c0163d24>] unmap_vmas+0x214/0x330
[<c0169adb>] exit_mmap+0xcb/0x2b0
[<c0127935>] mmput+0xb5/0x150
[<c012e062>] do_exit+0x1b2/0x960
[<c01215f0>] do_page_fault+0x0/0x56a
[<c01215f0>] do_page_fault+0x0/0x56a
[<c010d367>] die+0x227/0x230
[<c01217d6>] do_page_fault+0x1e6/0x56a
[<c0122d1e>] recalc_task_prio+0x8e/0x1b0
[<c0122f9a>] try_to_wake_up+0x15a/0x290
[<c01215f0>] do_page_fault+0x0/0x56a
[<c010cb79>] error_code+0x2d/0x38
[<c015ac3c>] free_block+0x4c/0x2f0
[<c015afba>] cache_flusharray+0xda/0x2b0
[<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
[<c018158c>] free_buffer_head+0x2c/0x60
[<c018158c>] free_buffer_head+0x2c/0x60
[<c0181353>] try_to_free_buffers+0x143/0x220
[<c0294bab>] linvfs_release_page+0x7b/0x80
[<c017f243>] try_to_release_page+0x53/0x60
[<c015fec7>] shrink_list+0x8b7/0xac0
[<c0123eef>] schedule+0x36f/0x8a0
[<c015e489>] __pagevec_release+0x29/0x40
[<c01602d5>] shrink_cache+0x205/0x610
[<c0127515>] autoremove_wake_function+0x25/0x50
[<c0161228>] shrink_zone+0x78/0xa0
[<c01612ee>] shrink_caches+0x9e/0xc0
[<c01613b7>] try_to_free_pages+0xa7/0x170
[<c015541a>] __alloc_pages+0x1fa/0x380
[<c0165c17>] do_anonymous_page+0xc7/0x520
[<c017a899>] do_sync_write+0x89/0xc0
[<c01668f2>] handle_mm_fault+0x132/0x310
[<c0121940>] do_page_fault+0x350/0x56a
[<c01698de>] do_brk+0x14e/0x230
[<c016790e>] sys_brk+0xee/0x120
[<c01215f0>] do_page_fault+0x0/0x56a
[<c010cb79>] error_code+0x2d/0x38
Call Trace:
[<c0124411>] schedule+0x891/0x8a0
[<c0163ae1>] unmap_page_range+0x41/0x70
[<c0163d24>] unmap_vmas+0x214/0x330
[<c0169adb>] exit_mmap+0xcb/0x2b0
[<c0127935>] mmput+0xb5/0x150
[<c012e062>] do_exit+0x1b2/0x960
[<c01215f0>] do_page_fault+0x0/0x56a
[<c01215f0>] do_page_fault+0x0/0x56a
[<c010d367>] die+0x227/0x230
[<c01217d6>] do_page_fault+0x1e6/0x56a
[<c0122d1e>] recalc_task_prio+0x8e/0x1b0
[<c0122f9a>] try_to_wake_up+0x15a/0x290
[<c01215f0>] do_page_fault+0x0/0x56a
[<c010cb79>] error_code+0x2d/0x38
[<c015ac3c>] free_block+0x4c/0x2f0
[<c015afba>] cache_flusharray+0xda/0x2b0
[<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
[<c018158c>] free_buffer_head+0x2c/0x60
[<c018158c>] free_buffer_head+0x2c/0x60
[<c0181353>] try_to_free_buffers+0x143/0x220
[<c0294bab>] linvfs_release_page+0x7b/0x80
[<c017f243>] try_to_release_page+0x53/0x60
[<c015fec7>] shrink_list+0x8b7/0xac0
[<c0123eef>] schedule+0x36f/0x8a0
[<c015e489>] __pagevec_release+0x29/0x40
[<c01602d5>] shrink_cache+0x205/0x610
[<c0127515>] autoremove_wake_function+0x25/0x50
[<c0161228>] shrink_zone+0x78/0xa0
[<c01612ee>] shrink_caches+0x9e/0xc0
[<c01613b7>] try_to_free_pages+0xa7/0x170
[<c015541a>] __alloc_pages+0x1fa/0x380
[<c0165c17>] do_anonymous_page+0xc7/0x520
[<c017a899>] do_sync_write+0x89/0xc0
[<c01668f2>] handle_mm_fault+0x132/0x310
[<c0121940>] do_page_fault+0x350/0x56a
[<c01698de>] do_brk+0x14e/0x230
[<c016790e>] sys_brk+0xee/0x120
[<c01215f0>] do_page_fault+0x0/0x56a
[<c010cb79>] error_code+0x2d/0x38
Call Trace:
[<c0124411>] schedule+0x891/0x8a0
[<c0163ae1>] unmap_page_range+0x41/0x70
[<c0163d24>] unmap_vmas+0x214/0x330
[<c0169adb>] exit_mmap+0xcb/0x2b0
[<c0127935>] mmput+0xb5/0x150
[<c012e062>] do_exit+0x1b2/0x960
[<c01215f0>] do_page_fault+0x0/0x56a
[<c01215f0>] do_page_fault+0x0/0x56a
[<c010d367>] die+0x227/0x230
[<c01217d6>] do_page_fault+0x1e6/0x56a
[<c0122d1e>] recalc_task_prio+0x8e/0x1b0
[<c0122f9a>] try_to_wake_up+0x15a/0x290
[<c01215f0>] do_page_fault+0x0/0x56a
[<c010cb79>] error_code+0x2d/0x38
[<c015ac3c>] free_block+0x4c/0x2f0
[<c015afba>] cache_flusharray+0xda/0x2b0
[<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
[<c018158c>] free_buffer_head+0x2c/0x60
[<c018158c>] free_buffer_head+0x2c/0x60
[<c0181353>] try_to_free_buffers+0x143/0x220
[<c0294bab>] linvfs_release_page+0x7b/0x80
[<c017f243>] try_to_release_page+0x53/0x60
[<c015fec7>] shrink_list+0x8b7/0xac0
[<c0123eef>] schedule+0x36f/0x8a0
[<c015e489>] __pagevec_release+0x29/0x40
[<c01602d5>] shrink_cache+0x205/0x610
[<c0127515>] autoremove_wake_function+0x25/0x50
[<c0161228>] shrink_zone+0x78/0xa0
[<c01612ee>] shrink_caches+0x9e/0xc0
[<c01613b7>] try_to_free_pages+0xa7/0x170
[<c015541a>] __alloc_pages+0x1fa/0x380
[<c0165c17>] do_anonymous_page+0xc7/0x520
[<c017a899>] do_sync_write+0x89/0xc0
[<c01668f2>] handle_mm_fault+0x132/0x310
[<c0121940>] do_page_fault+0x350/0x56a
[<c01698de>] do_brk+0x14e/0x230
[<c016790e>] sys_brk+0xee/0x120
[<c01215f0>] do_page_fault+0x0/0x56a
[<c010cb79>] error_code+0x2d/0x38
Call Trace:
[<c0126b1b>] __might_sleep+0xab/0xd0
[<c01677b9>] remove_shared_vm_struct+0x39/0xa0
[<c0169be1>] exit_mmap+0x1d1/0x2b0
[<c0127935>] mmput+0xb5/0x150
[<c012e062>] do_exit+0x1b2/0x960
[<c01215f0>] do_page_fault+0x0/0x56a
[<c01215f0>] do_page_fault+0x0/0x56a
[<c010d367>] die+0x227/0x230
[<c01217d6>] do_page_fault+0x1e6/0x56a
[<c0122d1e>] recalc_task_prio+0x8e/0x1b0
[<c0122f9a>] try_to_wake_up+0x15a/0x290
[<c01215f0>] do_page_fault+0x0/0x56a
[<c010cb79>] error_code+0x2d/0x38
[<c015ac3c>] free_block+0x4c/0x2f0
[<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
[<c018158c>] free_buffer_head+0x2c/0x60
[<c018158c>] free_buffer_head+0x2c/0x60
[<c0181353>] try_to_free_buffers+0x143/0x220
[<c0294bab>] linvfs_release_page+0x7b/0x80
[<c017f243>] try_to_release_page+0x53/0x60
[<c015fec7>] shrink_list+0x8b7/0xac0
[<c0123eef>] schedule+0x36f/0x8a0
[<c015e489>] __pagevec_release+0x29/0x40
[<c01602d5>] shrink_cache+0x205/0x610
[<c0127515>] autoremove_wake_function+0x25/0x50
[<c0161228>] shrink_zone+0x78/0xa0
[<c01612ee>] shrink_caches+0x9e/0xc0
[<c01613b7>] try_to_free_pages+0xa7/0x170
[<c015541a>] __alloc_pages+0x1fa/0x380
[<c0165c17>] do_anonymous_page+0xc7/0x520
[<c017a899>] do_sync_write+0x89/0xc0
[<c01668f2>] handle_mm_fault+0x132/0x310
[<c0121940>] do_page_fault+0x350/0x56a
[<c01698de>] do_brk+0x14e/0x230
[<c016790e>] sys_brk+0xee/0x120
[<c01215f0>] do_page_fault+0x0/0x56a
[<c010cb79>] error_code+0x2d/0x38
Warning (Oops_read): Code line not seen, dumping what data is available
Trace; c0124411 <schedule+891/8a0>
Trace; c0163ae1 <unmap_page_range+41/70>
Trace; c0163d24 <unmap_vmas+214/330>
Trace; c0169adb <exit_mmap+cb/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c0124411 <schedule+891/8a0>
Trace; c0163ae1 <unmap_page_range+41/70>
Trace; c0163d24 <unmap_vmas+214/330>
Trace; c0169adb <exit_mmap+cb/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c0124411 <schedule+891/8a0>
Trace; c0163ae1 <unmap_page_range+41/70>
Trace; c0163d24 <unmap_vmas+214/330>
Trace; c0169adb <exit_mmap+cb/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c0126b1b <__might_sleep+ab/d0>
Trace; c01677b9 <remove_shared_vm_struct+39/a0>
Trace; c0169be1 <exit_mmap+1d1/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
[email protected] wrote:
>
> kernel BUG at mm/slab.c:1957!
> invalid operand: 0000 [#1]
> CPU: 0
> EIP: 0060:[free_block+336/752] Not tainted
> EIP: 0060:[<c015ad40>] Not tainted
> Using defaults from ksymoops -t elf32-i386 -a i386
> EFLAGS: 00010096
> eax: 00000045 ebx: 00000006 ecx: c0693854 edx: c056e4f8
> esi: cd09a000 edi: cd09a018 ebp: cf821c68 esp: cf821c3c
> ds: 007b es: 007b ss: 0068
> Stack: c0502240 c0502e1d cd09af18 c0652a00 00000001 0000003a cd09af18 0000000f
> cffdef08 c4bcd180 00000010 cf821ca0 c015afba cffed800 cffdef08 00000010
> 00000282 c1161ca0 00000000 00000001 cffee730 00000010 00010c00 c4bcd180
> Call Trace:
> [<c015afba>] cache_flusharray+0xda/0x2b0
> [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
> [<c018158c>] free_buffer_head+0x2c/0x60
> [<c018158c>] free_buffer_head+0x2c/0x60
urgh, there are several reports of this and it's always the buffer_head
slab. The code in there is trivial so perhaps it's just that the large
number of buffer_heads makes them a fat target.
You should have also seen the message "slab: double free detected in cache
'buffer_head', objp 0xNNNNNNNN".
Don't know, sorry.
>Date: Wed, 19 Nov 2003 17:07:53 -0800
>De: Andrew Morton <[email protected]>
>A: [email protected]
>Copie ?: [email protected]
>Sujet: Re: [Oops] i386 mm/slab.c (cache_flusharray)
>
>[email protected] wrote:
>>
>> kernel BUG at mm/slab.c:1957!
[...]
>
>urgh, there are several reports of this and it's always the buffer_head
>slab. The code in there is trivial so perhaps it's just that the large
>number of buffer_heads makes them a fat target.
>
>You should have also seen the message "slab: double free detected in cache
>'buffer_head', objp 0xNNNNNNNN".
Yeah, you right, I forgot to mention it, it was just above the Oops in the logs:
---
slab: double free detected in cache 'buffer_head', objp cd09af18.
------------[ cut here ]------------
kernel BUG at mm/slab.c:1957!
---
>Don't know, sorry.
Is there any thing I can do to help figure out where does the problem comes from ? Anyway, thanks for your answer.
Regards,
Jerome Pinot
[email protected] wrote:
>
> kernel BUG at mm/slab.c:1957!
> ---
>
> >Don't know, sorry.
>
> Is there any thing I can do to help figure out where does the problem comes from ?
Well it's interesting that it is repeatable.
First thing to do is to eliminate hardware failures:
1: Is the oops always the same, or does the machine crash in other ways,
with different backtraces?
2: Try running memtest86 on that machine for 12 hours or more.
3: Can the problem be reproduced on other machines?
4: try a different compiler version.
Thanks.
----Message d'origine----
>Date: Wed, 19 Nov 2003 18:09:43 -0800
>De: Andrew Morton <[email protected]>
>A: [email protected]
>Copie ?: [email protected]
>Sujet: Re: [Oops] i386 mm/slab.c (cache_flusharray)
[...]
>> Is there any thing I can do to help figure out where does the problem comes from ?
>
>Well it's interesting that it is repeatable.
>First thing to do is to eliminate hardware failures:
>
>1: Is the oops always the same, or does the machine crash in other ways, with different backtraces?
Well, I can't check right know but seemed to be the same. I will keep the next 5 oops with same distro and make a diff to be sure.
>2: Try running memtest86 on that machine for 12 hours or more.
OK
>3: Can the problem be reproduced on other machines?
Unfortunately, I can't use any other computer for this (or I will lose some friends :-) If there was already some reports about this bug, it can be good to compare the hardware and/or environment with these others people.
>4: try a different compiler version.
I already tried gcc 3.2.3 and 3.3.1 (2.95.3 to be confirmed)
I will make the tests (1, 2 and confirmed 4) and give you the results tomorrow.
Just an idea: could it be an ACPI problem ?
I will try some boot parameters too, to be sure...
>Thanks.
Your welcome
Jerome Pinot
----Message d'origine----
>Date: Wed, 19 Nov 2003 18:09:43 -0800
>De: Andrew Morton <[email protected]>
>A: [email protected]
>Copie ?: [email protected]
>Sujet: Re: [Oops] i386 mm/slab.c (cache_flusharray)
[...]
>Well it's interesting that it is repeatable.
>
>First thing to do is to eliminate hardware failures:
>
>1: Is the oops always the same, or does the machine crash in other ways,
> with different backtraces?
>
>2: Try running memtest86 on that machine for 12 hours or more.
>
>3: Can the problem be reproduced on other machines?
>
>4: try a different compiler version.
First, some results about some tests (not finish yet)
0. Increase verbosity of the printk (thanks to Manfred):
(compilation of kernel)
slab: double free detected in cache 'buffer_head', objp c4c8e3d8, objnr 10,
slabp c4c8e000, s_mem c4c8e180, bufctl ffffffff.
(compilation of firebird)
slab: double free detected in cache 'pte_chain', objp c18a6600, objnr 10,
slabp c18a6000, s_mem c18a6100, bufctl ffffffff.
1. Reproductibility : yes, it oops each time I try to compile a kernel, for example, at around 75% of the task.
Always oops if I try to compile during a quite long time
One funny thing, though. I got one oops without freeze. After error of gcc, I went back to the shell but when I called ksymoops 2 commands later, everything freezed.
About the backtrace, well I'm not sure. Are you talking about what follow the `call trace` etc ? The problem is the system don't have always the time to flush everything to the log, I often got only the printk. But I always got the cache_flusharray thing in first position.
2. Test mem (not yet, I need some time). But as I said, I never had oops before, with 2.6.0-test from 4 to 9. I compiled all my LFS with 2.6.0-test9 vanilla without problem.
3. Change compiler: confirm same problem with gcc 2.95.3, 3.2.3, 3.3.1
x. ACPI: same oops with `acpi=off pci=noacpi` at boot
Summary: Oops reproductible when heavy load, bug in mm/slab.c
Don't have this problem with 2.6.0-test9 and prior
Problem appears in the last patches, before 15 november
(cset-20031115_0206) so I looked for something wrong.
I tried to remove some of the last patches (mm/ioremap.c,
mm/filemap.c, mm/memory.c) but still got oops.
Should be another patch. Which one else can I remove to test ?
Regards,
Jerome
--- 2.6/mm/slab.c 2003-11-18 18:18:20.000000000 +0100
+++ build-2.6/mm/slab.c 2003-11-21 19:50:02.000000000 +0100
@@ -153,9 +153,9 @@
* is less than 512 (PAGE_SIZE<<3), but greater than 256.
*/
-#define BUFCTL_END 0xffffFFFF
-#define BUFCTL_FREE 0xffffFFFE
-#define SLAB_LIMIT 0xffffFFFD
+#define BUFCTL_END 0xfeffFFFF
+#define BUFCTL_FREE 0xf7ffFFFE
+#define SLAB_LIMIT 0xf0ffFFFD
typedef unsigned int kmem_bufctl_t;
/* Max number of objs-per-slab for caches which use off-slab slabs.
On Fri, 21 Nov 2003 [email protected] wrote:
>
> Summary: Oops reproductible when heavy load, bug in mm/slab.c
Do you have CONFIG_PREEMPT on, and if so, does it go away if you compile
without PREEMPT? We have at least one other bug that seems to be dependent
on CONFIG_PREEMPT.
Linus
>> Summary: Oops reproductible when heavy load, bug in mm/slab.c
>
>Do you have CONFIG_PREEMPT on, and if so, does it go away if you compile
>without PREEMPT? We have at least one other bug that seems to be dependent
>on CONFIG_PREEMPT.
>
> Linus
Yes, I have CONFIG_PREEMPT=y in my .config
I will try without next time.
Here is the result about what asked Manfred.
In my logs, I found:
---
Nov 21 05:46:48 gegenux kernel: slab: double free detected in cache 'buffer_head', objp c4c8e3d8, objnr 10, slabp c4c8e000,
Nov 21 05:46:49 gegenux kernel: slab: double free detected in cache 'buffer_head', objp c9a582ac, objnr 5, slabp c9a58000,
Nov 21 07:01:50 gegenux kernel: slab: double free detected in cache 'pte_chain', objp c18a6600, objnr 10, slabp c18a6000,
---
So the objnr can be different but it's in the same oops (look the time). The slabp always finish by 0xXXXXX000.
I compiled again with the patch you gave.
First compilation (kernel) no freeze, simple error of `as` but I got this in the logs:
---
slab error in cache_free_debugcheck(): cache `bio': double free, or memory outside object was overwritten
Call Trace:
[kmem_cache_free+687/912] kmem_cache_free+0x2af/0x390
[<c015b8cf>] kmem_cache_free+0x2af/0x390
[mempool_free+224/544] mempool_free+0xe0/0x220
[<c01535d0>] mempool_free+0xe0/0x220
[mempool_free+224/544] mempool_free+0xe0/0x220
[<c01535d0>] mempool_free+0xe0/0x220
[kernel_map_pages+40/144] kernel_map_pages+0x28/0x90
[<c0122bd8>] kernel_map_pages+0x28/0x90
[bio_destructor+57/96] bio_destructor+0x39/0x60
[<c01816f9>] bio_destructor+0x39/0x60
[bio_put+41/64] bio_put+0x29/0x40
[<c01818e9>] bio_put+0x29/0x40
[end_bio_bh_io_sync+56/64] end_bio_bh_io_sync+0x38/0x40
[<c0180e18>] end_bio_bh_io_sync+0x38/0x40
[bio_endio+77/128] bio_endio+0x4d/0x80
--cut--
[background_writeout+0/176] background_writeout+0x0/0xb0
[<c01568a0>] background_writeout+0x0/0xb0
[kernel_thread_helper+5/12] kernel_thread_helper+0x5/0xc
[<c01092a9>] kernel_thread_helper+0x5/0xc
c6fd7870: redzone 1: 0x170fc2a5, redzone 2: 0x160fc2a5.
---
System looks OK, I tried a second compilation just after and this time I got an oops:
---
slab: double free detected in cache 'buffer_head', objp cc3f9798, objnr 26, slabp cc3f9000, s_mem cc3f9180 bufctl f7ffffff.
mm/slab.c:1777: spin_lock(mm/slab.c:cffed844) already locked by mm/slab.c/1994
---cut---
No modules in ksyms, skipping objects
No ksyms, skipping lsmod
kernel BUG at mm/slab.c:1956!
invalid operand: 0000 [#1]
CPU: 0
EIP: 0060:[free_block+363/784] Not tainted
EIP: 0060:[<c015ad6b>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010092
eax: 0000007f ebx: 0000000a ecx: c06973dc edx: c05712f8
esi: cc3f9000 edi: cc3f9018 ebp: cf821c68 esp: cf821c34
ds: 007b es: 007b ss: 0068
Stack: c0504d00 c05058fd cc3f9798 0000001a cc3f9000 cc3f9180 f7ffffff 0000001a
cc3f9798 0000000b cffdef08 c3bb8180 00000010 cf821ca0 c015afea cffed800
cffdef08 00000010 cf821ce8 c010cabc 80010c00 cd37e000 cffee730 00000010
Call Trace:
[<c015afea>] cache_flusharray+0xda/0x2b0
[<c010cabc>] common_interrupt+0x18/0x20
[<c015b7cd>] kmem_cache_free+0x1ad/0x39
---cut---
You can find the complete log here:
http://cercle-daejeon.homelinux.org/oops-log.txt
Hope this will help...
Jerome
[email protected] wrote:
>c6fd7870: redzone 1: 0x170fc2a5, redzone 2: 0x160fc2a5.
>
Single bit error: redzone 2 must be 0x170fc2a5.
>---
>System looks OK, I tried a second compilation just after and this time I got an oops:
>---
>slab: double free detected in cache 'buffer_head', objp cc3f9798, objnr 26, slabp cc3f9000, s_mem cc3f9180 bufctl f7ffffff.
>
>
Good.
+#define BUFCTL_END 0xfeffFFFF
+#define BUFCTL_FREE 0xf7ffFFFE
+#define SLAB_LIMIT 0xf0ffFFFD
f7ffffff is not a valid value, slab never writes that into a bufctl.
Someone did a ++ or "|= 1", or a hw bug.
I think the Athlon cpus have ECC for the L2 cache - could you check in
the bios that ECC checking is enabled?
--
Manfred
Sorry to be late,
>De: Linus Torvalds <[email protected]>
[...]
>> Summary: Oops reproductible when heavy load, bug in mm/slab.c
>
>Do you have CONFIG_PREEMPT on, and if so, does it go away if you compile without PREEMPT? We have at least one other bug that seems to be dependent on CONFIG_PREEMPT
I compiled without PREEMPT and first it seemed good. I could compile again a kernel without problem.
But later, I got the same oops when doing something else
(like `./configure` in parallele with a `make install` on other tty) so CONFIG_PREEMPT doesn't seem to be the cause, unfortunately, but a parameter than can affect the probability of getting the oops.
>De: Manfred Spraul <[email protected]>
[...]
>>slab: double free detected in cache 'buffer_head', objp cc3f9798, objnr 26, slabp cc3f9000, s_mem cc3f9180 bufctl f7ffffff.
>>
>Good.
>
>+#define BUFCTL_END 0xfeffFFFF
>+#define BUFCTL_FREE 0xf7ffFFFE
>+#define SLAB_LIMIT 0xf0ffFFFD
This seems to solve the problem, no oops during kernel compilation. Unfortunately, considering what I wrote just above, I'm not so sure it's really solved. Now I will use the 2.6.0-test10 and make again tests (alone, with this patch, with PREEMPT_CONFIG=n)
>f7ffffff is not a valid value, slab never writes that into a bufctl.
>Someone did a ++ or "|= 1", or a hw bug.
>I think the Athlon cpus have ECC for the L2 cache - could you check in
>the bios that ECC checking is enabled?
Well, cheap mainboard (VIA K7S5A) with cheap BIOS. I can only {en,dis}able cache L1/L2, nothing about ECC. DRAM is set to safe.
But as I said, I got no problem with 2.6.0-test9 vanilla. I compiled all my LFS/BLFS with it during several days.
I even use it these days as rescue kernel to compile the others.
Thanks for your help,
Jerome Pinot
Here are the results for test10 about the oops in slab.c
When I say `compilation` with no explanation, it means compilation of 2.6.0-test10 when using this same kernel, with my .config file (vmlinuz is 2.5M).
1. 2.6.0-test10 vanilla + PREEMPT_CONFIG=y + patch printk
Kernel oops during compilation, as for 2.6.0-test9 : at around 80% of the task.
---
slab: double free detected in cache 'buffer_head', objp cc2dbb58, objnr 42, slabp cc2db000, s_mem cc2db180, bufctl ffffffff.
------------[ cut here ]------------
kernel BUG at mm/slab.c:1956!
invalid operand: 0000 [#1]
CPU: 0
EIP: 0060:[free_block+357/784] Not tainted
EIP: 0060:[<c015ad55>] Not tainted
EFLAGS: 00010092
EIP is at free_block+0x165/0x310
eax: 00000080 ebx: 00000000 ecx: c0697854 edx: c05714f8
esi: cc2db000 edi: cc2db018 ebp: cf821c68 esp: cf821c34
ds: 007b es: 007b ss: 0068
Process kswapd0 (pid: 8, threadinfo=cf820000 task=cf849960)
Stack: c0504f40 c0505b3d cc2dbb58 0000002a cc2db000 cc2db180 ffffffff 0000002a
cc2dbb58 0000000d cffdef08 c9e93180 00000010 cf821ca0 c015afda cffed800
cffdef08 00000010 00000282 c11c89a0 00000000 00000001 cffee730 00000010
Call Trace:
[cache_flusharray+218/688] cache_flusharray+0xda/0x2b0
[<c015afda>] cache_flusharray+0xda/0x2b0
[kmem_cache_free+429/912] kmem_cache_free+0x1ad/0x390
---
full log can be found here: http://cercle-daejeon.homelinux.org/oops-full.txt
Cleaned oops (ksymoops): http://cercle-daejeon.homelinux.org/oops.txt
Config of the kernel : http://cercle-daejeon.homelinux.org/config.txt
NB: I don't get any oops if I compile the kernel with default settings (vmlinuz around 1.5M)
2. 2.6.0-test10 vanilla + PREEMPT_CONFIG=n + patch printk
Argh, oops at the speed of light in the beginning of compilation. Too fast to catch something in the logs.
This invalidates the first idea of bad effect of PREEMPT, it's exactly the contrary in this case.
Second try, compilation is OK, 1 times, 2 times and failed during the third time.
Again, no logs, but this time I wrote down the printk:
---
slab: double free detected in cache 'bio', objp c888fc28, objnr 42, slabp c888f000, s_mem c888f1000, bufctl ffffffff.
---
This case is not easily reproductible.
3. 2.6.0-test10 vanilla + PREEMPT_CONFIG=y + patch printk + patch magic numbers
The patch solves the problem, I can compile 4 times a kernel and do heavy work in parallele (load average around 1.2 during 2 hours) without any problems.
Conclusion: well, this confirms some facts for 2.6.0-test10:
- oops reproductible if PREEMPT_CONFIG=y each time heavy load.
The limit of load, for my system (AMD tbird 1.2GHz, 256Mo) is somewhere between the compilation of a kernel of 1.5M (default settings) and a kernel of 2.5M (custom settings). Always occurs at about 80% of compilation in this second case.
- oops occurs even if PREEMPT_CONFIG=n, but with no really reproductibility. It needs heavy load too, but it's not enough. System hangs really quickly, no logs.
Finally, I just recall the patches of Manfred used here (printk and magic number):
diff -Nru a/mm/slab.c b/mm/slab.c 2003-11-22 09:00:00 +0900
--- a/mm/slab.c 2003-11-22 08:43:02.189656536 +0900
+++ b/mm/slab.c 2003-11-22 08:45:44.158033600 +0900
@@ -1952,8 +1952,7 @@
check_slabp(cachep, slabp);
#if DEBUG
if (slab_bufctl(slabp)[objnr] != BUFCTL_FREE) {
- printk(KERN_ERR "slab: double free detected in cache '%s', objp %p.\n",
- cachep->name, objp);
+ printk(KERN_ERR "slab: double free detected in cache '%s', objp %p, objnr %d, slabp %p, s_mem %p, bufctl %x.\n", cachep->name, objp, objnr, slabp, slabp->s_mem, slab_bufctl(slabp)[objnr]);
BUG();
}
#endif
diff -Nru a/mm/slab.c b/mm/slab.c 2003-11-22 09:00:00 +0900
--- a/mm/slab.c 2003-11-22 08:43:02.189656536 +0900
+++ b/mm/slab.c 2003-11-22 08:45:44.158033600 +0900
@@ -153,9 +153,9 @@
* is less than 512 (PAGE_SIZE<<3), but greater than 256.
*/
-#define BUFCTL_END 0xffffFFFF
-#define BUFCTL_FREE 0xffffFFFE
-#define SLAB_LIMIT 0xffffFFFD
+#define BUFCTL_END 0xfeffFFFF
+#define BUFCTL_FREE 0xf7ffFFFF
+#define SLAB_LIMIT 0xf0ffFFFD
typedef unsigned int kmem_bufctl_t;
/* Max number of objs-per-slab for caches which use off-slab slabs.
Regards,
Jerome Pinot
On Tue, 25 Nov 2003 [email protected] wrote:
>
> 3. 2.6.0-test10 vanilla + PREEMPT_CONFIG=y + patch printk + patch magic numbers
> The patch solves the problem, I can compile 4 times a kernel and do heavy work in parallele (load average around 1.2 during 2 hours) without any problems.
Those magic numbers don't make any sense. In particular, SLAB_LIMIT is
clearly bogus both in the original version and in the "magic number
patch". The only place that uses SLAB_LIMIT is the code that decides how
many entries fit in one slab, and quite frankly, it makes no _sense_ to
have a SLAB_LIMIT that is big enough to be unsigned.
"SLAB_LIMIT" should be something in the few hundreds, maybe.
Manfred? What is the logic behind those nonsensical numbers?
Linus
Here is the result of test of 2.6.0-test10 with the printk patch in slab.c and this new patch for fork.c from Linus :
# --------------------------------------------
# 03/11/25 [email protected] 1.1487
# Fix error return on concurrent fork() with threaded exit()
# --------------------------------------------
Again the test is made in heavy load (compilation of kernel)
1st compilation: OK
2nd compilation straight after, oops :
---
slab: double free detected in cache 'vm_area_struct', objp cd4783f8, objnr 10, slabp cd478000, s_mem cd478100, bufctl ffffffff.
------------[ cut here ]------------
kernel BUG at mm/slab.c:1956!
invalid operand: 0000 [#1]
CPU: 0
EIP: 0060:[free_block+357/784] Not tainted
EIP: 0060:[<c015ad55>] Not tainted
EFLAGS: 00010096
EIP is at free_block+0x165/0x310
eax: 00000083 ebx: 00000009 ecx: c0697854 edx: c05714f8
esi: cd478000 edi: cd478018 ebp: ceaddb78 esp: ceaddb44
ds: 007b es: 007b ss: 0068
Process login (pid: 222, threadinfo=ceadc000 task=ceb0c960)
Stack: c0504f40 c0502370 cd4783f8 0000000a cd478000 cd478100 ffffffff 0000000a
cd4783f8 00000005 cffdff08 c5b29100 00000010 ceaddbb0 c015afda cffed980
cffdff08 00000010 c95d7ef8 c5b29234 cffd91dc ceaddbc4 cffee788 00000010
Call Trace:
[cache_flusharray+218/688] cache_flusharray+0xda/0x2b0
[<c015afda>] cache_flusharray+0xda/0x2b0
[kmem_cache_free+429/912] kmem_cache_free+0x1ad/0x390
[<c015b7bd>] kmem_cache_free+0x1ad/0x390
[exit_mmap+505/688] exit_mmap+0x1f9/0x2b0
---
Full log : http://cercle-daejeon.homelinux.org/oops-full2.txt
Ksymoops : http://cercle-daejeon.homelinux.org/oops2.txt
Sorry that is doesn't work.
Maybe the best way is to find which patch between test9 and test10 makes this happen but it takes me a really long time, I have no idea how to choose. I already tested the files in mm subfolder unsuccessfully.
As I already said, the problem appeared in the cset-20031115_0206.txt.gz.
Regards,
Jerome Pinot
Linus Torvalds wrote:
>On Tue, 25 Nov 2003 [email protected] wrote:
>
>
>>3. 2.6.0-test10 vanilla + PREEMPT_CONFIG=y + patch printk + patch magic numbers
>>The patch solves the problem, I can compile 4 times a kernel and do heavy work in parallele (load average around 1.2 during 2 hours) without any problems.
>>
>>
>
>Those magic numbers don't make any sense. In particular, SLAB_LIMIT is
>clearly bogus both in the original version and in the "magic number
>patch". The only place that uses SLAB_LIMIT is the code that decides how
>many entries fit in one slab, and quite frankly, it makes no _sense_ to
>have a SLAB_LIMIT that is big enough to be unsigned.
>
Object numbers (kmem_bufctl_t) are unsigned, but some values have a
special meaning:
"-1" is the magic value for end-of-list.
"-2" is the magic value for object in use.
All other values are valid object numbers. Right now object numbers are
unsigned int, but initially I considered unsigned char or unsigned
short. And then an explicit SLAB_LIMIT is necessary - with unsigned
char, the limit would be 253 objects per slab, which could be reached if
someone creates objects smaller than 16 bytes.
In Jerome's case, the debug checks noticed that the object-in-use
sentinel was not in the bufctl entry during free, instead there was a
"-1". There are several sources for the "-1": My initial guess was
either a bug in slab, or a bad memory cell (only one bit difference).
Thus I sent him a patch that changes multiple bits. Result: It remained
a single bit change, i.e it's proven that slab doesn't write BUFCTL_END
into the wrong slot.
--
Manfred
first, some news
2.6.0-test11 makes same oops during second compilation of kernel. The vanilla kernel with PREEMPT always oops the same way. No matter, it's always reproductible.
2.6.0-test11 + Manfred's patch doesn't hang but I found a slab error in the logs that occured during a compilation. (I didn't find this for -test10, I was lucky ?)
So, there is no more way for my system to run a kernel > -test9 without problem.
>De: Manfred Spraul <[email protected]>
[...]
>There are several sources for the "-1": My initial guess was either a bug in slab, or a bad memory cell (only one bit difference).
>Thus I sent him a patch that changes multiple bits. Result: It remained a single bit change, i.e it's proven that slab doesn't write BUFCTL_END into the wrong slot.
Thanks for your explanation.
Should I try with L1 and/or L2 cache disable on my computer (I don't know if it's safe) ?
I trust my hardware but it's better to get some facts.
Jerome Pinot
(between LFS/BLFS, kernel compilation and tests compilation, I will surely break kind of record about load average :-)
[email protected] wrote:
>Thanks for your explanation.
>Should I try with L1 and/or L2 cache disable on my computer (I don't know if it's safe) ?
>I trust my hardware but it's better to get some facts.
>
No, it wouldn't help. Something in the kernel randomly corrupts memory.
I'm certain that it's not slab. I'm also fairly certain that it's not
the hardware - IBM guys reproduced corruptions on both ppc64 and i386
systems (bugzilla 1097 and 1497). The corrupted object is the slab
structure or the bufctl entries - data near the beginning of a page. But
I have no idea how to pinpoint it.
--
Manfred
I triggered the slab oops with a very small kernel -test11 (~700KB):
---
slab: double free detected in cache 'biovec-1', objp c12c2df0, objn 122, slabp c12c2000, s_mem c12c2280, bufctl ffffffff
---
Oops occurs very quickly during compilation.
Here is the .config file (I removed unset options). Full config can be found at http://cercle-daejeon.homelinux.org/misc/config-small.txt , if you want diff with other config for example.
BTW, I don't understand why I can't remove config about game port, mouse and DOS partition. Compilation without DMA fails.
I will reduce it again by removing FB and try unset some options in "linux for embbeded systems" and in debug.
Hope it can help find the problem.
Jerome Pinot
---
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_CLEAN_COMPILE=y
CONFIG_STANDALONE=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_LOG_BUF_SHIFT=14
CONFIG_KALLSYMS=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_X86_PC=y
CONFIG_M386=y
CONFIG_X86_L1_CACHE_SHIFT=4
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_X86_PPRO_FENCE=y
CONFIG_X86_F00F_BUG=y
CONFIG_NOHIGHMEM=y
CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_BINFMT_ELF=y
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_BLK_DEV_GENERIC=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_BLK_DEV_ADMA=y
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_INPUT=y
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_SOUND_GAMEPORT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_FB=y
CONFIG_FB_VESA=y
CONFIG_VIDEO_SELECT=y
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_PCI_CONSOLE=y
CONFIG_FONTS=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_XFS_FS=y
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_RAMFS=y
CONFIG_MSDOS_PARTITION=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_SLAB=y
CONFIG_DEBUG_IOVIRT=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_SPINLOCK_SLEEP=y
CONFIG_FRAME_POINTER=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_PC=y
---