Hi all,
I have something really strange while running kernel 2.6.19.
See the following lines:
Dec 5 23:50:49 kernel: do_get_write_access: OOM for frozen_buffer
Dec 5 23:50:49 kernel: ext3_free_blocks_sb: aborting transaction: Out of
memory in __ext3_journal_get_undo_access
Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in ext3_free_blocks_sb:
Out of memory
Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
ext3_reserve_inode_write: Readonly filesystem
Dec 5 23:50:50 kernel: EXT3-fs error (device md1) in ext3_truncate: Out
of memory
Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
ext3_reserve_inode_write: Readonly filesystem
Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in ext3_orphan_del:
Readonly filesystem
Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
ext3_reserve_inode_write: Readonly filesystem
Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in ext3_delete_inode:
Out of memory
And three days later the same:
Dec 8 08:24:29 kernel: do_get_write_access: OOM for frozen_buffer
Dec 8 08:24:29 kernel: ext3_reserve_inode_write: aborting transaction:
Out of memory in __ext3_journal_get_write_access
Dec 8 08:24:29 kernel: EXT3-fs error (device md1) in
ext3_reserve_inode_write: Out of memory
Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in ext3_dirty_inode:
Out of memory
Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in ext3_new_blocks:
Readonly filesystem
Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in
ext3_reserve_inode_write: Readonly filesystem
Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in ext3_dirty_inode:
Out of memory
Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in ext3_prepare_write:
Out of memory
Now the funny thing is, with kernel 2.6.18.3 I did not had these errors.
Could it be my memory that is just going nuts, or something else? I have
seen some other topics about the EXT3 corruption problems. Maybe this is
also the same thing?
Thanks for reading.
Hi,
> I have something really strange while running kernel 2.6.19.
>
> See the following lines:
>
> Dec 5 23:50:49 kernel: do_get_write_access: OOM for frozen_buffer
> Dec 5 23:50:49 kernel: ext3_free_blocks_sb: aborting transaction: Out of
> memory in __ext3_journal_get_undo_access
> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in ext3_free_blocks_sb:
> Out of memory
> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
> ext3_reserve_inode_write: Readonly filesystem
> Dec 5 23:50:50 kernel: EXT3-fs error (device md1) in ext3_truncate: Out
> of memory
> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
> ext3_reserve_inode_write: Readonly filesystem
> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in ext3_orphan_del:
> Readonly filesystem
> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
> ext3_reserve_inode_write: Readonly filesystem
> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in ext3_delete_inode:
> Out of memory
>
> And three days later the same:
>
> Dec 8 08:24:29 kernel: do_get_write_access: OOM for frozen_buffer
> Dec 8 08:24:29 kernel: ext3_reserve_inode_write: aborting transaction:
> Out of memory in __ext3_journal_get_write_access
> Dec 8 08:24:29 kernel: EXT3-fs error (device md1) in
> ext3_reserve_inode_write: Out of memory
> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in ext3_dirty_inode:
> Out of memory
> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in ext3_new_blocks:
> Readonly filesystem
> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in
> ext3_reserve_inode_write: Readonly filesystem
> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in ext3_dirty_inode:
> Out of memory
> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in ext3_prepare_write:
> Out of memory
>
> Now the funny thing is, with kernel 2.6.18.3 I did not had these errors.
> Could it be my memory that is just going nuts, or something else? I have
> seen some other topics about the EXT3 corruption problems. Maybe this is
> also the same thing?
Probably some error on the kernel's side. Maybe somebody (e.g. my new
code in jbd commit) forgets to release some buffers? What does your
/proc/slabinfo look like? Thanks for report.
Honza
Hi there,
Here is the output of /proc/slabinfo
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab>
<pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata
<active_slabs> <num_slabs> <sharedavail>
ip_conntrack_expect 0 0 136 28 1 : tunables 120 60
8 : slabdata 0 0 0
ip_conntrack 641 1040 304 13 1 : tunables 54 27 8
: slabdata 80 80 0
fib6_nodes 5 59 64 59 1 : tunables 120 60 8
: slabdata 1 1 0
ip6_dst_cache 4 12 320 12 1 : tunables 54 27 8
: slabdata 1 1 0
ndisc_cache 1 12 320 12 1 : tunables 54 27 8
: slabdata 1 1 0
RAWv6 5 7 1088 7 2 : tunables 24 12 8
: slabdata 1 1 0
UDPv6 1 7 1088 7 2 : tunables 24 12 8
: slabdata 1 1 0
tw_sock_TCPv6 39 100 192 20 1 : tunables 120 60 8
: slabdata 5 5 0
request_sock_TCPv6 0 0 192 20 1 : tunables 120 60
8 : slabdata 0 0 0
TCPv6 12 12 1920 2 1 : tunables 24 12 8
: slabdata 6 6 0
rpc_buffers 8 8 2048 2 1 : tunables 24 12 8
: slabdata 4 4 0
rpc_tasks 8 10 384 10 1 : tunables 54 27 8
: slabdata 1 1 0
rpc_inode_cache 6 8 960 4 1 : tunables 54 27 8
: slabdata 2 2 0
ip_fib_alias 12 59 64 59 1 : tunables 120 60 8
: slabdata 1 1 0
ip_fib_hash 11 59 64 59 1 : tunables 120 60 8
: slabdata 1 1 0
jbd_1k 0 0 1024 4 1 : tunables 54 27 8
: slabdata 0 0 0
dm_tio 0 0 24 144 1 : tunables 120 60 8
: slabdata 0 0 0
dm_io 0 0 40 92 1 : tunables 120 60 8
: slabdata 0 0 0
uhci_urb_priv 0 0 56 67 1 : tunables 120 60 8
: slabdata 0 0 0
jbd_4k 5 10 4096 1 1 : tunables 24 12 8
: slabdata 5 10 0
ext3_inode_cache 22447 22696 968 4 1 : tunables 54 27 8
: slabdata 5674 5674 0
ext3_xattr 0 0 88 44 1 : tunables 120 60 8
: slabdata 0 0 0
journal_handle 24 144 24 144 1 : tunables 120 60 8
: slabdata 1 1 0
journal_head 740 1280 96 40 1 : tunables 120 60 8
: slabdata 27 32 480
revoke_table 4 202 16 202 1 : tunables 120 60 8
: slabdata 1 1 0
revoke_record 16 112 32 112 1 : tunables 120 60 8
: slabdata 1 1 0
UNIX 60 92 896 4 1 : tunables 54 27 8
: slabdata 23 23 0
flow_cache 0 0 128 30 1 : tunables 120 60 8
: slabdata 0 0 0
msi_cache 5 59 64 59 1 : tunables 120 60 8
: slabdata 1 1 0
cfq_ioc_pool 0 0 160 24 1 : tunables 120 60 8
: slabdata 0 0 0
cfq_pool 0 0 160 24 1 : tunables 120 60 8
: slabdata 0 0 0
mqueue_inode_cache 1 7 1088 7 2 : tunables 24 12
8 : slabdata 1 1 0
isofs_inode_cache 0 0 768 5 1 : tunables 54 27 8
: slabdata 0 0 0
hugetlbfs_inode_cache 1 5 752 5 1 : tunables 54 27
8 : slabdata 1 1 0
ext2_inode_cache 0 0 920 4 1 : tunables 54 27 8
: slabdata 0 0 0
ext2_xattr 0 0 88 44 1 : tunables 120 60 8
: slabdata 0 0 0
dnotify_cache 1 92 40 92 1 : tunables 120 60 8
: slabdata 1 1 0
dquot 0 0 256 15 1 : tunables 120 60 8
: slabdata 0 0 0
eventpoll_pwq 1 53 72 53 1 : tunables 120 60 8
: slabdata 1 1 0
eventpoll_epi 1 20 192 20 1 : tunables 120 60 8
: slabdata 1 1 0
inotify_event_cache 0 0 40 92 1 : tunables 120 60
8 : slabdata 0 0 0
inotify_watch_cache 0 0 72 53 1 : tunables 120 60
8 : slabdata 0 0 0
kioctx 0 0 384 10 1 : tunables 54 27 8
: slabdata 0 0 0
kiocb 0 0 256 15 1 : tunables 120 60 8
: slabdata 0 0 0
fasync_cache 0 0 24 144 1 : tunables 120 60 8
: slabdata 0 0 0
shmem_inode_cache 220 224 960 4 1 : tunables 54 27 8
: slabdata 56 56 0
posix_timers_cache 0 0 152 25 1 : tunables 120 60
8 : slabdata 0 0 0
uid_cache 19 60 128 30 1 : tunables 120 60 8
: slabdata 2 2 0
ip_mrt_cache 0 0 128 30 1 : tunables 120 60 8
: slabdata 0 0 0
tcp_bind_bucket 25 112 32 112 1 : tunables 120 60 8
: slabdata 1 1 0
inet_peer_cache 8 59 64 59 1 : tunables 120 60 8
: slabdata 1 1 0
secpath_cache 0 0 64 59 1 : tunables 120 60 8
: slabdata 0 0 0
xfrm_dst_cache 0 0 384 10 1 : tunables 54 27 8
: slabdata 0 0 0
ip_dst_cache 1344 1344 320 12 1 : tunables 54 27 8
: slabdata 112 112 0
arp_cache 2 15 256 15 1 : tunables 120 60 8
: slabdata 1 1 0
RAW 3 4 896 4 1 : tunables 54 27 8
: slabdata 1 1 0
UDP 10 16 896 4 1 : tunables 54 27 8
: slabdata 4 4 0
tw_sock_TCP 4 20 192 20 1 : tunables 120 60 8
: slabdata 1 1 0
request_sock_TCP 16 30 128 30 1 : tunables 120 60 8
: slabdata 1 1 0
TCP 18 18 1792 2 1 : tunables 24 12 8
: slabdata 9 9 0
blkdev_ioc 57 134 56 67 1 : tunables 120 60 8
: slabdata 2 2 0
blkdev_queue 21 24 1672 4 2 : tunables 24 12 8
: slabdata 6 6 0
blkdev_requests 45 182 280 14 1 : tunables 54 27 8
: slabdata 13 13 0
biovec-256 7 7 4096 1 1 : tunables 24 12 8
: slabdata 7 7 0
biovec-128 7 8 2048 2 1 : tunables 24 12 8
: slabdata 4 4 0
biovec-64 7 8 1024 4 1 : tunables 54 27 8
: slabdata 2 2 0
biovec-16 7 15 256 15 1 : tunables 120 60 8
: slabdata 1 1 0
biovec-4 7 59 64 59 1 : tunables 120 60 8
: slabdata 1 1 0
biovec-1 207 808 16 202 1 : tunables 120 60 8
: slabdata 4 4 144
bio 397 1440 128 30 1 : tunables 120 60 8
: slabdata 48 48 48
sock_inode_cache 136 160 832 4 1 : tunables 54 27 8
: slabdata 40 40 0
skbuff_fclone_cache 99 161 512 7 1 : tunables 54 27
8 : slabdata 23 23 0
skbuff_head_cache 150 150 256 15 1 : tunables 120 60 8
: slabdata 10 10 0
file_lock_cache 61 80 192 20 1 : tunables 120 60 8
: slabdata 4 4 0
Acpi-Operand 1946 2006 64 59 1 : tunables 120 60 8
: slabdata 34 34 0
Acpi-ParseExt 0 0 64 59 1 : tunables 120 60 8
: slabdata 0 0 0
Acpi-Parse 0 0 40 92 1 : tunables 120 60 8
: slabdata 0 0 0
Acpi-State 0 0 80 48 1 : tunables 120 60 8
: slabdata 0 0 0
Acpi-Namespace 955 1008 32 112 1 : tunables 120 60 8
: slabdata 9 9 0
proc_inode_cache 406 485 752 5 1 : tunables 54 27 8
: slabdata 97 97 0
sigqueue 24 24 160 24 1 : tunables 120 60 8
: slabdata 1 1 0
radix_tree_node 8034 8106 536 7 1 : tunables 54 27 8
: slabdata 1158 1158 108
bdev_cache 27 32 960 4 1 : tunables 54 27 8
: slabdata 8 8 0
sysfs_dir_cache 3219 3264 80 48 1 : tunables 120 60 8
: slabdata 68 68 0
mnt_cache 28 30 256 15 1 : tunables 120 60 8
: slabdata 2 2 0
inode_cache 776 905 720 5 1 : tunables 54 27 8
: slabdata 181 181 0
dentry_cache 25160 25160 224 17 1 : tunables 120 60 8
: slabdata 1480 1480 0
filp 1079 1668 320 12 1 : tunables 54 27 8
: slabdata 139 139 47
names_cache 5 5 4096 1 1 : tunables 24 12 8
: slabdata 5 5 0
avc_node 12 53 72 53 1 : tunables 120 60 8
: slabdata 1 1 0
selinux_inode_security 1250 1280 96 40 1 : tunables 120 60
8 : slabdata 32 32 0
key_jar 22 60 192 20 1 : tunables 120 60 8
: slabdata 3 3 0
idr_layer_cache 64 70 528 7 1 : tunables 54 27 8
: slabdata 10 10 0
buffer_head 56406 95386 104 37 1 : tunables 120 60 8
: slabdata 2578 2578 480
mm_struct 85 104 960 4 1 : tunables 54 27 8
: slabdata 26 26 0
vm_area_struct 8170 10098 176 22 1 : tunables 120 60 8
: slabdata 459 459 354
fs_cache 71 180 128 30 1 : tunables 120 60 8
: slabdata 6 6 0
files_cache 86 121 704 11 2 : tunables 54 27 8
: slabdata 11 11 0
signal_cache 116 140 768 5 1 : tunables 54 27 8
: slabdata 28 28 0
sighand_cache 110 126 2112 3 2 : tunables 24 12 8
: slabdata 42 42 0
task_struct 130 136 1840 2 1 : tunables 24 12 8
: slabdata 68 68 0
anon_vma 954 1196 40 92 1 : tunables 120 60 8
: slabdata 13 13 0
pid 118 236 64 59 1 : tunables 120 60 8
: slabdata 4 4 0
shared_policy_node 0 0 48 77 1 : tunables 120 60
8 : slabdata 0 0 0
numa_policy 7 144 24 144 1 : tunables 120 60 8
: slabdata 1 1 0
size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0
: slabdata 0 0 0
size-131072 0 0 131072 1 32 : tunables 8 4 0
: slabdata 0 0 0
size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0
: slabdata 0 0 0
size-65536 1 1 65536 1 16 : tunables 8 4 0
: slabdata 1 1 0
size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0
: slabdata 0 0 0
size-32768 1 1 32768 1 8 : tunables 8 4 0
: slabdata 1 1 0
size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0
: slabdata 0 0 0
size-16384 8 10 16384 1 4 : tunables 8 4 0
: slabdata 8 10 0
size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0
: slabdata 0 0 0
size-8192 28 28 8192 1 2 : tunables 8 4 0
: slabdata 28 28 0
size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 8
: slabdata 0 0 0
size-4096 108 108 4096 1 1 : tunables 24 12 8
: slabdata 108 108 0
size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 8
: slabdata 0 0 0
size-2048 288 288 2048 2 1 : tunables 24 12 8
: slabdata 144 144 0
size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8
: slabdata 0 0 0
size-1024 341 400 1024 4 1 : tunables 54 27 8
: slabdata 100 100 27
size-512(DMA) 0 0 512 8 1 : tunables 54 27 8
: slabdata 0 0 0
size-512 522 536 512 8 1 : tunables 54 27 8
: slabdata 67 67 0
size-256(DMA) 0 0 256 15 1 : tunables 120 60 8
: slabdata 0 0 0
size-256 76 105 256 15 1 : tunables 120 60 8
: slabdata 7 7 0
size-192(DMA) 0 0 192 20 1 : tunables 120 60 8
: slabdata 0 0 0
size-192 1057 1120 192 20 1 : tunables 120 60 8
: slabdata 56 56 0
size-128(DMA) 0 0 128 30 1 : tunables 120 60 8
: slabdata 0 0 0
size-64(DMA) 0 0 64 59 1 : tunables 120 60 8
: slabdata 0 0 0
size-32(DMA) 0 0 32 112 1 : tunables 120 60 8
: slabdata 0 0 0
size-32 1969 2128 32 112 1 : tunables 120 60 8
: slabdata 19 19 0
size-128 1192 2010 128 30 1 : tunables 120 60 8
: slabdata 67 67 30
size-64 14943 15222 64 59 1 : tunables 120 60 8
: slabdata 258 258 0
kmem_cache 133 135 704 5 1 : tunables 54 27 8
: slabdata 27 27 0
Greetings,
Jim van Wel.
> Hi,
>
>> I have something really strange while running kernel 2.6.19.
>>
>> See the following lines:
>>
>> Dec 5 23:50:49 kernel: do_get_write_access: OOM for frozen_buffer
>> Dec 5 23:50:49 kernel: ext3_free_blocks_sb: aborting transaction: Out
>> of
>> memory in __ext3_journal_get_undo_access
>> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
>> ext3_free_blocks_sb:
>> Out of memory
>> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
>> ext3_reserve_inode_write: Readonly filesystem
>> Dec 5 23:50:50 kernel: EXT3-fs error (device md1) in ext3_truncate: Out
>> of memory
>> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> ext3_reserve_inode_write: Readonly filesystem
>> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in ext3_orphan_del:
>> Readonly filesystem
>> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> ext3_reserve_inode_write: Readonly filesystem
>> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in ext3_delete_inode:
>> Out of memory
>>
>> And three days later the same:
>>
>> Dec 8 08:24:29 kernel: do_get_write_access: OOM for frozen_buffer
>> Dec 8 08:24:29 kernel: ext3_reserve_inode_write: aborting transaction:
>> Out of memory in __ext3_journal_get_write_access
>> Dec 8 08:24:29 kernel: EXT3-fs error (device md1) in
>> ext3_reserve_inode_write: Out of memory
>> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in ext3_dirty_inode:
>> Out of memory
>> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in ext3_new_blocks:
>> Readonly filesystem
>> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in
>> ext3_reserve_inode_write: Readonly filesystem
>> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in ext3_dirty_inode:
>> Out of memory
>> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in
>> ext3_prepare_write:
>> Out of memory
>>
>> Now the funny thing is, with kernel 2.6.18.3 I did not had these errors.
>> Could it be my memory that is just going nuts, or something else? I have
>> seen some other topics about the EXT3 corruption problems. Maybe this is
>> also the same thing?
> Probably some error on the kernel's side. Maybe somebody (e.g. my new
> code in jbd commit) forgets to release some buffers? What does your
> /proc/slabinfo look like? Thanks for report.
>
> Honza
>
Hi,
> Here is the output of /proc/slabinfo
>
> slabinfo - version: 2.1
> # name <active_objs> <num_objs> <objsize> <objperslab>
> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata
> <active_slabs> <num_slabs> <sharedavail>
> jbd_4k 5 10 4096 1 1 : tunables 24 12 8
> : slabdata 5 10 0
> ext3_inode_cache 22447 22696 968 4 1 : tunables 54 27 8
> : slabdata 5674 5674 0
> journal_head 740 1280 96 40 1 : tunables 120 60 8
> : slabdata 27 32 480
> buffer_head 56406 95386 104 37 1 : tunables 120 60 8
> : slabdata 2578 2578 480
<snip>
Thanks for the info. I should have said I'm interested about
/proc/slabinfo close to the state when the machine starts complaining
about OOM. Your current numbers look quite sensible so it's hard to see
if we really leak something or not...
> >> I have something really strange while running kernel 2.6.19.
> >>
> >> See the following lines:
> >>
> >> Dec 5 23:50:49 kernel: do_get_write_access: OOM for frozen_buffer
> >> Dec 5 23:50:49 kernel: ext3_free_blocks_sb: aborting transaction: Out
> >> of
> >> memory in __ext3_journal_get_undo_access
> >> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
> >> ext3_free_blocks_sb:
> >> Out of memory
> >> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
> >> ext3_reserve_inode_write: Readonly filesystem
> >> Dec 5 23:50:50 kernel: EXT3-fs error (device md1) in ext3_truncate: Out
> >> of memory
> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
> >> ext3_reserve_inode_write: Readonly filesystem
> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in ext3_orphan_del:
> >> Readonly filesystem
> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
> >> ext3_reserve_inode_write: Readonly filesystem
> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in ext3_delete_inode:
> >> Out of memory
> >>
> >> And three days later the same:
> >>
> >> Dec 8 08:24:29 kernel: do_get_write_access: OOM for frozen_buffer
> >> Dec 8 08:24:29 kernel: ext3_reserve_inode_write: aborting transaction:
> >> Out of memory in __ext3_journal_get_write_access
> >> Dec 8 08:24:29 kernel: EXT3-fs error (device md1) in
> >> ext3_reserve_inode_write: Out of memory
> >> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in ext3_dirty_inode:
> >> Out of memory
> >> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in ext3_new_blocks:
> >> Readonly filesystem
> >> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in
> >> ext3_reserve_inode_write: Readonly filesystem
> >> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in ext3_dirty_inode:
> >> Out of memory
> >> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in
> >> ext3_prepare_write:
> >> Out of memory
> >>
> >> Now the funny thing is, with kernel 2.6.18.3 I did not had these errors.
> >> Could it be my memory that is just going nuts, or something else? I have
> >> seen some other topics about the EXT3 corruption problems. Maybe this is
> >> also the same thing?
> > Probably some error on the kernel's side. Maybe somebody (e.g. my new
> > code in jbd commit) forgets to release some buffers? What does your
> > /proc/slabinfo look like? Thanks for report.
Honza
Hi there,
Well, that's kind of difficult because it looks a little random when he
does it, and a interval of three days, but also is maybe random.
And the most difficult part is it's only for three seconds, and than it's
gone, so making a crontab script every minute might not even notice it?
Or am I just making a mistake here?
Thanks!
Jim.
> Hi,
>
>> Here is the output of /proc/slabinfo
>>
>> slabinfo - version: 2.1
>> # name <active_objs> <num_objs> <objsize> <objperslab>
>> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata
>> <active_slabs> <num_slabs> <sharedavail>
>> jbd_4k 5 10 4096 1 1 : tunables 24 12
>> 8
>> : slabdata 5 10 0
>> ext3_inode_cache 22447 22696 968 4 1 : tunables 54 27
>> 8
>> : slabdata 5674 5674 0
>> journal_head 740 1280 96 40 1 : tunables 120 60
>> 8
>> : slabdata 27 32 480
>> buffer_head 56406 95386 104 37 1 : tunables 120 60
>> 8
>> : slabdata 2578 2578 480
> <snip>
>
> Thanks for the info. I should have said I'm interested about
> /proc/slabinfo close to the state when the machine starts complaining
> about OOM. Your current numbers look quite sensible so it's hard to see
> if we really leak something or not...
>
>> >> I have something really strange while running kernel 2.6.19.
>> >>
>> >> See the following lines:
>> >>
>> >> Dec 5 23:50:49 kernel: do_get_write_access: OOM for frozen_buffer
>> >> Dec 5 23:50:49 kernel: ext3_free_blocks_sb: aborting transaction:
>> Out
>> >> of
>> >> memory in __ext3_journal_get_undo_access
>> >> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
>> >> ext3_free_blocks_sb:
>> >> Out of memory
>> >> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
>> >> ext3_reserve_inode_write: Readonly filesystem
>> >> Dec 5 23:50:50 kernel: EXT3-fs error (device md1) in ext3_truncate:
>> Out
>> >> of memory
>> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> >> ext3_reserve_inode_write: Readonly filesystem
>> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> ext3_orphan_del:
>> >> Readonly filesystem
>> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> >> ext3_reserve_inode_write: Readonly filesystem
>> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> ext3_delete_inode:
>> >> Out of memory
>> >>
>> >> And three days later the same:
>> >>
>> >> Dec 8 08:24:29 kernel: do_get_write_access: OOM for frozen_buffer
>> >> Dec 8 08:24:29 kernel: ext3_reserve_inode_write: aborting
>> transaction:
>> >> Out of memory in __ext3_journal_get_write_access
>> >> Dec 8 08:24:29 kernel: EXT3-fs error (device md1) in
>> >> ext3_reserve_inode_write: Out of memory
>> >> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in
>> ext3_dirty_inode:
>> >> Out of memory
>> >> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in
>> ext3_new_blocks:
>> >> Readonly filesystem
>> >> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in
>> >> ext3_reserve_inode_write: Readonly filesystem
>> >> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in
>> ext3_dirty_inode:
>> >> Out of memory
>> >> Dec 8 08:24:32 kernel: EXT3-fs error (device md1) in
>> >> ext3_prepare_write:
>> >> Out of memory
>> >>
>> >> Now the funny thing is, with kernel 2.6.18.3 I did not had these
>> errors.
>> >> Could it be my memory that is just going nuts, or something else? I
>> have
>> >> seen some other topics about the EXT3 corruption problems. Maybe this
>> is
>> >> also the same thing?
>> > Probably some error on the kernel's side. Maybe somebody (e.g. my
>> new
>> > code in jbd commit) forgets to release some buffers? What does your
>> > /proc/slabinfo look like? Thanks for report.
>
> Honza
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
Hi,
> Well, that's kind of difficult because it looks a little random when he
> does it, and a interval of three days, but also is maybe random.
>
> And the most difficult part is it's only for three seconds, and than it's
> gone, so making a crontab script every minute might not even notice it?
Oh, so the machine does not crash or go totally out of memory when this
happens? At least it seems the filesystem is remounted RO?
<snip>
> >> >> Dec 5 23:50:49 kernel: do_get_write_access: OOM for frozen_buffer
> >> >> Dec 5 23:50:49 kernel: ext3_free_blocks_sb: aborting transaction:
> >> Out
> >> >> of
> >> >> memory in __ext3_journal_get_undo_access
> >> >> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
> >> >> ext3_free_blocks_sb:
> >> >> Out of memory
> >> >> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
> >> >> ext3_reserve_inode_write: Readonly filesystem
> >> >> Dec 5 23:50:50 kernel: EXT3-fs error (device md1) in ext3_truncate:
> >> Out
> >> >> of memory
> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
> >> >> ext3_reserve_inode_write: Readonly filesystem
> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
> >> ext3_orphan_del:
> >> >> Readonly filesystem
> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
> >> >> ext3_reserve_inode_write: Readonly filesystem
> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
> >> ext3_delete_inode:
> >> >> Out of memory
<snip>
Honza
--
Jan Kara <[email protected]>
SuSE CR Labs
Hi,
> Oh, so the machine does not crash or go totally out of memory when this
> happens? At least it seems the filesystem is remounted RO?
Well, that's the most funny thing. It doesn't remount RO, it's still
mounted as RW.
Like here:
/dev/md1 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/md0 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/proc on /var/named/chroot/proc type none (rw,bind)
So it doesn't mount it to RO. And yes the system is just up and running.
It's a server. So it's on for 24/7. It now only happened for two times
when running the kernel 2.6.19. So it even looks like these messages are
false, because in all logs, I don't see Segfaults anywhere. And there
should be some if the error is really true what is said.
Anyway, I cannot trace back this easily as you can see.
So what can I do to help with these strange messages?
Thanks!
Jim.
> Hi,
>
>> Well, that's kind of difficult because it looks a little random when he
>> does it, and a interval of three days, but also is maybe random.
>>
>> And the most difficult part is it's only for three seconds, and than
>> it's
>> gone, so making a crontab script every minute might not even notice it?
> Oh, so the machine does not crash or go totally out of memory when this
> happens? At least it seems the filesystem is remounted RO?
>
> <snip>
>> >> >> Dec 5 23:50:49 kernel: do_get_write_access: OOM for frozen_buffer
>> >> >> Dec 5 23:50:49 kernel: ext3_free_blocks_sb: aborting transaction:
>> >> Out
>> >> >> of
>> >> >> memory in __ext3_journal_get_undo_access
>> >> >> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
>> >> >> ext3_free_blocks_sb:
>> >> >> Out of memory
>> >> >> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
>> >> >> ext3_reserve_inode_write: Readonly filesystem
>> >> >> Dec 5 23:50:50 kernel: EXT3-fs error (device md1) in
>> ext3_truncate:
>> >> Out
>> >> >> of memory
>> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> >> >> ext3_reserve_inode_write: Readonly filesystem
>> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> >> ext3_orphan_del:
>> >> >> Readonly filesystem
>> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> >> >> ext3_reserve_inode_write: Readonly filesystem
>> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> >> ext3_delete_inode:
>> >> >> Out of memory
> <snip>
>
> Honza
> --
> Jan Kara <[email protected]>
> SuSE CR Labs
>
Hi there,
Well, after I was getting the error again, I now switched back to kernel
2.6.18.1, and going to check if I am getting the same errors. I'll keep
you posted about the progress. If a week have passed and no errors has
shown, I'll e-mail this again.
Thanks!
Jim.
> Hi,
>
>> Well, that's kind of difficult because it looks a little random when he
>> does it, and a interval of three days, but also is maybe random.
>>
>> And the most difficult part is it's only for three seconds, and than
>> it's
>> gone, so making a crontab script every minute might not even notice it?
> Oh, so the machine does not crash or go totally out of memory when this
> happens? At least it seems the filesystem is remounted RO?
>
> <snip>
>> >> >> Dec 5 23:50:49 kernel: do_get_write_access: OOM for frozen_buffer
>> >> >> Dec 5 23:50:49 kernel: ext3_free_blocks_sb: aborting transaction:
>> >> Out
>> >> >> of
>> >> >> memory in __ext3_journal_get_undo_access
>> >> >> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
>> >> >> ext3_free_blocks_sb:
>> >> >> Out of memory
>> >> >> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
>> >> >> ext3_reserve_inode_write: Readonly filesystem
>> >> >> Dec 5 23:50:50 kernel: EXT3-fs error (device md1) in
>> ext3_truncate:
>> >> Out
>> >> >> of memory
>> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> >> >> ext3_reserve_inode_write: Readonly filesystem
>> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> >> ext3_orphan_del:
>> >> >> Readonly filesystem
>> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> >> >> ext3_reserve_inode_write: Readonly filesystem
>> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> >> ext3_delete_inode:
>> >> >> Out of memory
> <snip>
>
> Honza
> --
> Jan Kara <[email protected]>
> SuSE CR Labs
>
Hi all,
As promised, I should notify if no problems would happen. Well, no
problems happened. So when I was running kernel version 2.6.19 the
problems of my previous e-mails did happen, with kernel 2.6.18.1 running
right now, just none of the errors are comming up, and is running
smoothly.
Greetings,
Jim.
> Hi,
>
>> Well, that's kind of difficult because it looks a little random when he
>> does it, and a interval of three days, but also is maybe random.
>>
>> And the most difficult part is it's only for three seconds, and than
>> it's
>> gone, so making a crontab script every minute might not even notice it?
> Oh, so the machine does not crash or go totally out of memory when this
> happens? At least it seems the filesystem is remounted RO?
>
> <snip>
>> >> >> Dec 5 23:50:49 kernel: do_get_write_access: OOM for frozen_buffer
>> >> >> Dec 5 23:50:49 kernel: ext3_free_blocks_sb: aborting transaction:
>> >> Out
>> >> >> of
>> >> >> memory in __ext3_journal_get_undo_access
>> >> >> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
>> >> >> ext3_free_blocks_sb:
>> >> >> Out of memory
>> >> >> Dec 5 23:50:49 kernel: EXT3-fs error (device md1) in
>> >> >> ext3_reserve_inode_write: Readonly filesystem
>> >> >> Dec 5 23:50:50 kernel: EXT3-fs error (device md1) in
>> ext3_truncate:
>> >> Out
>> >> >> of memory
>> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> >> >> ext3_reserve_inode_write: Readonly filesystem
>> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> >> ext3_orphan_del:
>> >> >> Readonly filesystem
>> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> >> >> ext3_reserve_inode_write: Readonly filesystem
>> >> >> Dec 5 23:50:51 kernel: EXT3-fs error (device md1) in
>> >> ext3_delete_inode:
>> >> >> Out of memory
> <snip>
>
> Honza
> --
> Jan Kara <[email protected]>
> SuSE CR Labs
>