Hi list,
This happened yesterday on our 4-way system with highmem:
ksymoops 0.7c on i686 2.4.17-rc1. Options used
-v /usr/src/linux/vmlinux (specified)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.17-rc1/ (default)
-m /usr/src/linux/System.map (specified)
Warning (compare_maps): mismatch on symbol partition_name , ksyms_base
says c01ee900, vmlinux says c0156eb0. Ignoring ksyms_base entry
Dec 26 09:06:00 castor kernel: invalid operand: 0000
Dec 26 09:06:00 castor kernel: CPU: 0
Dec 26 09:06:00 castor kernel: EIP: 0010:[__free_pages_ok+75/508] Not tainted
Dec 26 09:06:00 castor kernel: EFLAGS: 00010202
Dec 26 09:06:00 castor kernel: eax: 00000840 ebx: c1e1dec0 ecx: c1e1dec0 edx: 00000000
Dec 26 09:06:00 castor kernel: esi: fe2b6fdd edi: 00000000 ebp: 00000000 esp: f1d79ef4
Dec 26 09:06:00 castor kernel: ds: 0018 es: 0018 ss: 0018
Dec 26 09:06:00 castor kernel: Process ps (pid: 32579, stackpage=f1d79000)
Dec 26 09:06:00 castor kernel: Stack: 00000360 fe2b6fdd 00000000 d3778000 e979a700 bffffc7d 00000000 00000000
Dec 26 09:06:00 castor kernel: 00000360 c012e8d8 c011de64 e979a700 00000000 00000360 d8124000 f1d79f44
Dec 26 09:06:00 castor kernel: f1d78000 e979a71c d3778000 e979a700 c1e1dec0 f4c973e0 c0152319 d8124000
Dec 26 09:06:00 castor kernel: Call Trace: [__free_pages+28/32] [access_process_vm+368/432] [proc_pid_environ+85/104]
[proc_info_read+89/296] [sys_read+142/196]
Dec 26 09:06:00 castor kernel: Code: 0f 0b 8b 43 18 a8 80 74 02 0f 0b 80
63 18 eb be 00 e0 ff ff
Using defaults from ksymoops -t elf32-i386 -a i386
Code; 00000000 Before first symbol
00000000 <_EIP>:
Code; 00000000 Before first symbol
0: 0f 0b ud2a
Code; 00000002 Before first symbol
2: 8b 43 18 mov 0x18(%ebx),%eax
Code; 00000005 Before first symbol
5: a8 80 test $0x80,%al
Code; 00000007 Before first symbol
7: 74 02 je b <_EIP+0xb> 0000000b Before
first symbol
Code; 00000009 Before first symbol
9: 0f 0b ud2a
Code; 0000000b Before first symbol
b: 80 63 18 eb andb $0xeb,0x18(%ebx)
Code; 0000000f Before first symbol
f: be 00 e0 ff ff mov $0xffffe000,%esi
1 warning issued. Results may not be reliable.
This box is a heavily loaded mail server, running what once was redhat
6.2. When users started complaining, it was around 13h and loadavg was
~400 and still growing. I couldn't even list the processes, hard reset was
the only option to put things back into normal quickly. Luckily I have ext3 :)
Any ideas?
--
Jure Pecar
Huh, it just happened again. Same symptoms, shell responsive, can do dmesg & such, but w or ps ax gives no reply. Load steadily increases.
ksymoops 0.7c on i686 2.4.17-rc1. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.17-rc1/ (default)
-m /usr/src/linux/System.map (default)
Warning: You did not tell me where to find symbol information. I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc. ksymoops -h explains the options.
Warning (compare_maps): mismatch on symbol partition_name , ksyms_base says c01ee900, System.map says c0156eb0. Ignoring ksyms_base entry
invalid operand: 0000
CPU: 2
EIP: 0010:[<c012e1cb>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010202
eax: 00000840 ebx: c2cda940 ecx: c2cda940 edx: 00000000
esi: fe068fdd edi: 00000000 ebp: 00000000 esp: c7f75ef4
ds: 0018 es: 0018 ss: 0018
Process ps (pid: 13231, stackpage=c7f75000)
Stack: 0000012e fe068fdd 00000000 ed2ad000 f75006c0 bffffeaf 00000000 00000000
0000012e c012e8d8 c011de64 f75006c0 00000000 0000012e f74b8000 c7f75f44
c7f74000 f75006dc ed2ad000 f75006c0 c2cda940 f74fed20 c0152319 f74b8000
Call Trace: [<c012e8d8>] [<c011de64>] [<c0152319>] [<c0152631>] [<c0134bee>]
[<c0106de3>]
Code: 0f 0b 8b 43 18 a8 80 74 02 0f 0b 80 63 18 eb be 00 e0 ff ff
>>EIP; c012e1cb <__free_pages_ok+4b/1fc> <=====
Trace; c012e8d8 <__free_pages+1c/20>
Trace; c011de64 <access_process_vm+170/1b0>
Trace; c0152319 <proc_pid_environ+55/68>
Trace; c0152631 <proc_info_read+59/128>
Trace; c0134bee <sys_read+8e/c4>
Trace; c0106de3 <system_call+2f/34>
Code; c012e1cb <__free_pages_ok+4b/1fc>
00000000 <_EIP>:
Code; c012e1cb <__free_pages_ok+4b/1fc> <=====
0: 0f 0b ud2a <=====
Code; c012e1cd <__free_pages_ok+4d/1fc>
2: 8b 43 18 mov 0x18(%ebx),%eax
Code; c012e1d0 <__free_pages_ok+50/1fc>
5: a8 80 test $0x80,%al
Code; c012e1d2 <__free_pages_ok+52/1fc>
7: 74 02 je b <_EIP+0xb> c012e1d6 <__free_pages_ok+56/1fc>
Code; c012e1d4 <__free_pages_ok+54/1fc>
9: 0f 0b ud2a
Code; c012e1d6 <__free_pages_ok+56/1fc>
b: 80 63 18 eb andb $0xeb,0x18(%ebx)
Code; c012e1da <__free_pages_ok+5a/1fc>
f: be 00 e0 ff ff mov $0xffffe000,%esi
2 warnings issued. Results may not be reliable.
--
Jure Pecar