2010-08-15 05:25:47

by M. Vefa Bicakci

[permalink] [raw]
Subject: [Bisected Regression in 2.6.35] A full tmpfs filesystem causes hibernationto hang

Hello all,

I am using Debian Sid on a Toshiba Satellite A100 laptop. After testing
2.6.35 for a while, I noticed that sometimes my hibernation attempts
would fail. I should say that I never had such a problem before 2.6.35.
The hibernation process hangs with 2.6.35 after printing the following:

=== 8< ===
...
Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.
PM: Preallocating image memory...
=== >8 ===

After a short investigation, I found out that this only happens when my
tmpfs filesystem on /tmp had a lot of data in it. When my tmpfs is empty,
I have no problems.

So I wrote a short script which fills up the tmpfs on /tmp and tries to
hibernate, and I bisected the kernel using this script.

The end result is that the following commit causes this regression:

=== 8< ===
commit bb21c7ce18eff8e6e7877ca1d06c6db719376e3c
Author: KOSAKI Motohiro <[email protected]>
Date: Fri Jun 4 14:15:05 2010 -0700

vmscan: fix do_try_to_free_pages() return value when priority==0 reclaim failure

...
=== >8 ===

I have run 2.6.35-rc6, 2.6.35 and 2.6.35.1 with this commit reverted,
and I am happy to say that I haven't experienced any problems for at
least 17 days.

It looks like this change was included with 2.6.35-rc1. I am sorry
for not testing earlier.

I am willing to do testing in case anyone would like me to try patches.

Regards,

M. Vefa Bicakci


2010-08-17 02:37:44

by KOSAKI Motohiro

[permalink] [raw]
Subject: Re: [Bisected Regression in 2.6.35] A full tmpfs filesystem causes hibernationto hang

> Hello all,
>
> I am using Debian Sid on a Toshiba Satellite A100 laptop. After testing
> 2.6.35 for a while, I noticed that sometimes my hibernation attempts
> would fail. I should say that I never had such a problem before 2.6.35.
> The hibernation process hangs with 2.6.35 after printing the following:
>
> === 8< ===
> ...
> Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.
> PM: Preallocating image memory...
> === >8 ===
>
> After a short investigation, I found out that this only happens when my
> tmpfs filesystem on /tmp had a lot of data in it. When my tmpfs is empty,
> I have no problems.
>
> So I wrote a short script which fills up the tmpfs on /tmp and tries to
> hibernate, and I bisected the kernel using this script.
>
> The end result is that the following commit causes this regression:
>
> === 8< ===
> commit bb21c7ce18eff8e6e7877ca1d06c6db719376e3c
> Author: KOSAKI Motohiro <[email protected]>
> Date: Fri Jun 4 14:15:05 2010 -0700
>
> vmscan: fix do_try_to_free_pages() return value when priority==0 reclaim failure
>
> ...
> === >8 ===
>
> I have run 2.6.35-rc6, 2.6.35 and 2.6.35.1 with this commit reverted,
> and I am happy to say that I haven't experienced any problems for at
> least 17 days.
>
> It looks like this change was included with 2.6.35-rc1. I am sorry
> for not testing earlier.
>
> I am willing to do testing in case anyone would like me to try patches.

Wow.

I'm very surprised this report because 1) the above commit changed
do_try_to_free_pages() return value 2) but current hibernation code is
ignoring this return value. Hmm... I have to investigate this very
interesting issue.

Thanks this report, and please give me a bit time.


2010-08-22 11:06:32

by KOSAKI Motohiro

[permalink] [raw]
Subject: Re: [Bisected Regression in 2.6.35] A full tmpfs filesystem causes hibernationto hang

> Hello all,
>
> I am using Debian Sid on a Toshiba Satellite A100 laptop. After testing
> 2.6.35 for a while, I noticed that sometimes my hibernation attempts
> would fail. I should say that I never had such a problem before 2.6.35.
> The hibernation process hangs with 2.6.35 after printing the following:
>
> === 8< ===
> ...
> Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.
> PM: Preallocating image memory...
> === >8 ===
>
> After a short investigation, I found out that this only happens when my
> tmpfs filesystem on /tmp had a lot of data in it. When my tmpfs is empty,
> I have no problems.
>
> So I wrote a short script which fills up the tmpfs on /tmp and tries to
> hibernate, and I bisected the kernel using this script.
>
> The end result is that the following commit causes this regression:
>
> === 8< ===
> commit bb21c7ce18eff8e6e7877ca1d06c6db719376e3c
> Author: KOSAKI Motohiro <[email protected]>
> Date: Fri Jun 4 14:15:05 2010 -0700
>
> vmscan: fix do_try_to_free_pages() return value when priority==0 reclaim failure
>
> ...
> === >8 ===
>
> I have run 2.6.35-rc6, 2.6.35 and 2.6.35.1 with this commit reverted,
> and I am happy to say that I haven't experienced any problems for at
> least 17 days.
>
> It looks like this change was included with 2.6.35-rc1. I am sorry
> for not testing earlier.
>
> I am willing to do testing in case anyone would like me to try patches.
>
> Regards,
>
> M. Vefa Bicakci

Hmm...
I've tested hibernation case for a while. but I have no luck. I couldn't
reproduce your issue. Very sorry. Can you please help our debugging?
If possible, I hope to run following three test.

1. Please let me know your machine & test script

% cat /proc/meminfo
% cat /proc/vmstat
% cat /proc/zoneinfo
% df
% cat your-fills-up-the-tmpfs-script

2. call shrink_all_memory() forcely and show result

% cat /proc/meminfo
% cat /proc/zoneinfo
# echo 1 > /proc/sys/vm/shrink_all_memory
# tail /var/log/messages
% cat /proc/meminfo
% cat /proc/zoneinfo


3. reset zone_reclaim_stat and rerun shrink_all_memory

# echo 1 > /proc/sys/vm/reset_reclaim_stat
% cat /proc/meminfo
% cat /proc/zoneinfo
# echo 1 > /proc/sys/vm/shrink_all_memory
# tail /var/log/messages
% cat /proc/meminfo
% cat /proc/zoneinfo



Attachments:
0001-debug-print-sysctl.patch (2.12 kB)
0003-debug-reset_reclaim_stat.patch (1.62 kB)
0004-debug-zone_info-reclaim_stat.patch (1.54 kB)
Download all attachments

2010-08-22 16:29:04

by M. Vefa Bicakci

[permalink] [raw]
Subject: Re: [Bisected Regression in 2.6.35] A full tmpfs filesystem causeshibernation to hang

Hello,

> I've tested hibernation case for a while. but I have no luck. I couldn't
> reproduce your issue. Very sorry. Can you please help our debugging?
> If possible, I hope to run following three test.

First of all, thanks a lot for spending time on this regression
I have been experiencing. I really appreciate it.

Sorry to hear that you weren't able to reproduce the issue. Well the
good (or bad?) news is that I am able to reproduce it with 2.6.35.3
with your patches applied.

I should note that after applying your patches and trying a hibernation
with a full tmpfs, a printk prints extra information on the screen just
before the hibernation process hangs. The last time I ran it, it printed:

=== 8< ===
shrink_all_memory: req: 342067 reclaimed: 27062 free: 340221
=== >8 ===

A piece of information that may be relevant or irrelevant is that my
swap space is on a dm-crypt volume.

Appended are the results of the tests you asked me to carry out.
If you'd like, I can send in private a tarball containing this
information in separate files.

Once again, thanks a lot for helping out.

M. Vefa Bicakci

>
> 1. Please let me know your machine & test script

Please note that I filled up the tmpfs filesystem between step 1
and step 2.

>
> % cat /proc/meminfo

MemTotal: 3104484 kB
MemFree: 2817616 kB
Buffers: 31156 kB
Cached: 142124 kB
SwapCached: 0 kB
Active: 116464 kB
Inactive: 137424 kB
Active(anon): 80852 kB
Inactive(anon): 24820 kB
Active(file): 35612 kB
Inactive(file): 112604 kB
Unevictable: 32 kB
Mlocked: 32 kB
HighTotal: 2226632 kB
HighFree: 1994008 kB
LowTotal: 877852 kB
LowFree: 823608 kB
SwapTotal: 1999540 kB
SwapFree: 1999540 kB
Dirty: 116 kB
Writeback: 0 kB
AnonPages: 80636 kB
Mapped: 43768 kB
Shmem: 25068 kB
Slab: 15120 kB
SReclaimable: 7516 kB
SUnreclaim: 7604 kB
KernelStack: 1856 kB
PageTables: 2420 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3551780 kB
Committed_AS: 337784 kB
VmallocTotal: 122880 kB
VmallocUsed: 16308 kB
VmallocChunk: 94312 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 20472 kB
DirectMap4M: 888832 kB


> % cat /proc/vmstat

nr_free_pages 704435
nr_inactive_anon 6199
nr_active_anon 20213
nr_inactive_file 28153
nr_active_file 8904
nr_unevictable 8
nr_mlock 8
nr_anon_pages 20159
nr_mapped 10943
nr_file_pages 43317
nr_dirty 2
nr_writeback 0
nr_slab_reclaimable 1879
nr_slab_unreclaimable 1900
nr_page_table_pages 605
nr_kernel_stack 232
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 6261
pgpgin 150947
pgpgout 3984
pswpin 0
pswpout 0
pgalloc_dma 2
pgalloc_normal 55622
pgalloc_high 219709
pgalloc_movable 0
pgfree 980252
pgactivate 12703
pgdeactivate 0
pgfault 584903
pgmajfault 1077
pgrefill_dma 0
pgrefill_normal 0
pgrefill_high 0
pgrefill_movable 0
pgsteal_dma 0
pgsteal_normal 0
pgsteal_high 0
pgsteal_movable 0
pgscan_kswapd_dma 0
pgscan_kswapd_normal 0
pgscan_kswapd_high 0
pgscan_kswapd_movable 0
pgscan_direct_dma 0
pgscan_direct_normal 0
pgscan_direct_high 0
pgscan_direct_movable 0
pginodesteal 0
slabs_scanned 0
kswapd_steal 0
kswapd_inodesteal 0
kswapd_low_wmark_hit_quickly 0
kswapd_high_wmark_hit_quickly 0
kswapd_skip_congestion_wait 0
pageoutrun 0
allocstall 0
pgrotated 0
htlb_buddy_alloc_success 0
htlb_buddy_alloc_fail 0
unevictable_pgs_culled 7828
unevictable_pgs_scanned 0
unevictable_pgs_rescued 7042
unevictable_pgs_mlocked 8558
unevictable_pgs_munlocked 8550
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
unevictable_pgs_mlockfreed 0

> % cat /proc/zoneinfo

Node 0, zone DMA
pages free 2614
min 16
low 20
high 24
scanned 0
spanned 4080
present 3951
nr_free_pages 2614
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 0
nr_active_file 0
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 0
nr_file_pages 0
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 2
nr_page_table_pages 0
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 865, 3022, 3022)
pagesets
cpu: 0
count: 0
high: 0
batch: 1
vm stats threshold: 4
cpu: 1
count: 0
high: 0
batch: 1
vm stats threshold: 4
all_unreclaimable: 0
prev_priority: 12
start_pfn: 16
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 0
recent_file_scanned: 0
anon_ratio: 30
Node 0, zone Normal
pages free 203226
min 932
low 1165
high 1398
scanned 0
spanned 223230
present 221486
nr_free_pages 203226
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 5727
nr_active_file 2143
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 1
nr_file_pages 7869
nr_dirty 9
nr_writeback 0
nr_slab_reclaimable 1880
nr_slab_unreclaimable 1898
nr_page_table_pages 0
nr_kernel_stack 232
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 0, 17259, 17259)
pagesets
cpu: 0
count: 127
high: 186
batch: 31
vm stats threshold: 16
cpu: 1
count: 61
high: 186
batch: 31
vm stats threshold: 16
all_unreclaimable: 0
prev_priority: 12
start_pfn: 4096
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 2330
recent_file_scanned: 11064
anon_ratio: 8
Node 0, zone HighMem
pages free 498378
min 128
low 709
high 1290
scanned 0
spanned 556658
present 552309
nr_free_pages 498378
nr_inactive_anon 6283
nr_active_anon 20225
nr_inactive_file 22446
nr_active_file 6830
nr_unevictable 8
nr_mlock 8
nr_anon_pages 20171
nr_mapped 10962
nr_file_pages 35621
nr_dirty 1
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 0
nr_page_table_pages 605
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 6345
protection: (0, 0, 0, 0)
pagesets
cpu: 0
count: 155
high: 186
batch: 31
vm stats threshold: 24
cpu: 1
count: 154
high: 186
batch: 31
vm stats threshold: 24
all_unreclaimable: 0
prev_priority: 12
start_pfn: 227326
inactive_ratio: 4
recent_anon_rotated: 140735
recent_anon_scanned: 188078
recent_file_rotated: 9673
recent_file_scanned: 43796
anon_ratio: 11

> % df

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda15 10317828 7308360 2485352 75% /
tmpfs 1552240 8 1552232 1% /lib/init/rw
udev 1549148 220 1548928 1% /dev
tmpfs 1552240 4 1552236 1% /dev/shm
tmpfs 1552240 32 1552208 1% /tmp

> % cat your-fills-up-the-tmpfs-script

#!/bin/bash

dd if=/dev/zero of=/tmp/tmp bs=1024k count=1600
shred -vn1 /tmp/tmp

exit 0 ### removed when testing hibernation

read -p "Press enter to hibernate..."

echo -n reboot > /sys/power/disk
echo -n disk > /sys/power/state

echo "Exit code: $?"

rm -vf /tmp/tmp


>
> 2. call shrink_all_memory() forcely and show result

> # ./script.sh # Fill up the tmpfs

> % df # My extra step (I modified the script to fill only 1400MB.)

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda15 10317828 7308408 2485304 75% /
tmpfs 1552240 8 1552232 1% /lib/init/rw
udev 1549148 220 1548928 1% /dev
tmpfs 1552240 4 1552236 1% /dev/shm
tmpfs 1552240 1435036 117204 93% /tmp

>
> % cat /proc/meminfo

MemTotal: 3104484 kB
MemFree: 1339564 kB
Buffers: 42824 kB
Cached: 1595432 kB
SwapCached: 0 kB
Active: 1557540 kB
Inactive: 161944 kB
Active(anon): 1515072 kB
Inactive(anon): 25144 kB
Active(file): 42468 kB
Inactive(file): 136800 kB
Unevictable: 32 kB
Mlocked: 32 kB
HighTotal: 2226632 kB
HighFree: 538992 kB
LowTotal: 877852 kB
LowFree: 800572 kB
SwapTotal: 1999540 kB
SwapFree: 1999540 kB
Dirty: 20 kB
Writeback: 0 kB
AnonPages: 81256 kB
Mapped: 44032 kB
Shmem: 1458992 kB
Slab: 26312 kB
SReclaimable: 18632 kB
SUnreclaim: 7680 kB
KernelStack: 1816 kB
PageTables: 2424 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3551780 kB
Committed_AS: 1772060 kB
VmallocTotal: 122880 kB
VmallocUsed: 16308 kB
VmallocChunk: 94312 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 20472 kB
DirectMap4M: 888832 kB

> % cat /proc/zoneinfo

Node 0, zone DMA
pages free 2614
min 16
low 20
high 24
scanned 0
spanned 4080
present 3951
nr_free_pages 2614
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 0
nr_active_file 0
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 0
nr_file_pages 0
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 2
nr_page_table_pages 0
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 865, 3022, 3022)
pagesets
cpu: 0
count: 0
high: 0
batch: 1
vm stats threshold: 4
cpu: 1
count: 0
high: 0
batch: 1
vm stats threshold: 4
all_unreclaimable: 0
prev_priority: 12
start_pfn: 16
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 0
recent_file_scanned: 0
anon_ratio: 30
Node 0, zone Normal
pages free 197498
min 932
low 1165
high 1398
scanned 0
spanned 223230
present 221486
nr_free_pages 197498
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 7293
nr_active_file 3466
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 1
nr_file_pages 10758
nr_dirty 4
nr_writeback 0
nr_slab_reclaimable 4658
nr_slab_unreclaimable 1918
nr_page_table_pages 0
nr_kernel_stack 227
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 0, 17259, 17259)
pagesets
cpu: 0
count: 157
high: 186
batch: 31
vm stats threshold: 16
cpu: 1
count: 81
high: 186
batch: 31
vm stats threshold: 16
all_unreclaimable: 0
prev_priority: 12
start_pfn: 4096
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 3652
recent_file_scanned: 15280
anon_ratio: 9
Node 0, zone HighMem
pages free 134717
min 128
low 709
high 1290
scanned 0
spanned 556658
present 552309
nr_free_pages 134717
nr_inactive_anon 6261
nr_active_anon 378769
nr_inactive_file 26907
nr_active_file 7152
nr_unevictable 8
nr_mlock 8
nr_anon_pages 20315
nr_mapped 11007
nr_file_pages 398782
nr_dirty 2
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 0
nr_page_table_pages 606
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 364723
protection: (0, 0, 0, 0)
pagesets
cpu: 0
count: 136
high: 186
batch: 31
vm stats threshold: 24
cpu: 1
count: 182
high: 186
batch: 31
vm stats threshold: 24
all_unreclaimable: 0
prev_priority: 12
start_pfn: 227326
inactive_ratio: 4
recent_anon_rotated: 912909
recent_anon_scanned: 1757064
recent_file_rotated: 10006
recent_file_scanned: 48952
anon_ratio: 14

> # echo 1 > /proc/sys/vm/shrink_all_memory

> # tail /var/log/messages

[ 472.454344] shrink_all_memory start
[ 472.454731] shrink_all_memory: req:1 reclaimed:29 free:334891

> % cat /proc/meminfo

MemTotal: 3104484 kB
MemFree: 1339556 kB
Buffers: 42864 kB
Cached: 1595252 kB
SwapCached: 0 kB
Active: 1557108 kB
Inactive: 162296 kB
Active(anon): 1514628 kB
Inactive(anon): 25540 kB
Active(file): 42480 kB
Inactive(file): 136756 kB
Unevictable: 32 kB
Mlocked: 32 kB
HighTotal: 2226632 kB
HighFree: 538992 kB
LowTotal: 877852 kB
LowFree: 800564 kB
SwapTotal: 1999540 kB
SwapFree: 1999540 kB
Dirty: 24 kB
Writeback: 0 kB
AnonPages: 81324 kB
Mapped: 44048 kB
Shmem: 1458884 kB
Slab: 26312 kB
SReclaimable: 18632 kB
SUnreclaim: 7680 kB
KernelStack: 1824 kB
PageTables: 2424 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3551780 kB
Committed_AS: 1771928 kB
VmallocTotal: 122880 kB
VmallocUsed: 16308 kB
VmallocChunk: 94312 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 20472 kB
DirectMap4M: 888832 kB


> % cat /proc/zoneinfo

Node 0, zone DMA
pages free 2614
min 16
low 20
high 24
scanned 0
spanned 4080
present 3951
nr_free_pages 2614
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 0
nr_active_file 0
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 0
nr_file_pages 0
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 2
nr_page_table_pages 0
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 865, 3022, 3022)
pagesets
cpu: 0
count: 0
high: 0
batch: 1
vm stats threshold: 4
cpu: 1
count: 0
high: 0
batch: 1
vm stats threshold: 4
all_unreclaimable: 0
prev_priority: 10
start_pfn: 16
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 0
recent_file_scanned: 0
anon_ratio: 30
Node 0, zone Normal
pages free 197527
min 932
low 1165
high 1398
scanned 0
spanned 223230
present 221486
nr_free_pages 197527
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 7304
nr_active_file 3466
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 1
nr_file_pages 10770
nr_dirty 5
nr_writeback 0
nr_slab_reclaimable 4657
nr_slab_unreclaimable 1918
nr_page_table_pages 0
nr_kernel_stack 228
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 0, 17259, 17259)
pagesets
cpu: 0
count: 157
high: 186
batch: 31
vm stats threshold: 16
cpu: 1
count: 67
high: 186
batch: 31
vm stats threshold: 16
all_unreclaimable: 0
prev_priority: 10
start_pfn: 4096
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 456
recent_file_scanned: 1918
anon_ratio: 9
Node 0, zone HighMem
pages free 134779
min 128
low 709
high 1290
scanned 0
spanned 556658
present 552309
nr_free_pages 134779
nr_inactive_anon 6389
nr_active_anon 378650
nr_inactive_file 26887
nr_active_file 7154
nr_unevictable 8
nr_mlock 8
nr_anon_pages 20331
nr_mapped 11014
nr_file_pages 398764
nr_dirty 2
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 0
nr_page_table_pages 606
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 364723
protection: (0, 0, 0, 0)
pagesets
cpu: 0
count: 144
high: 186
batch: 31
vm stats threshold: 24
cpu: 1
count: 115
high: 186
batch: 31
vm stats threshold: 24
all_unreclaimable: 0
prev_priority: 10
start_pfn: 227326
inactive_ratio: 4
recent_anon_rotated: 117656
recent_anon_scanned: 228960
recent_file_rotated: 1252
recent_file_scanned: 6163
anon_ratio: 14

>
>
> 3. reset zone_reclaim_stat and rerun shrink_all_memory
>
> # echo 1 > /proc/sys/vm/reset_reclaim_stat
> % cat /proc/meminfo

MemTotal: 3104484 kB
MemFree: 1339316 kB
Buffers: 42924 kB
Cached: 1595344 kB
SwapCached: 0 kB
Active: 1557224 kB
Inactive: 162344 kB
Active(anon): 1514632 kB
Inactive(anon): 25532 kB
Active(file): 42592 kB
Inactive(file): 136812 kB
Unevictable: 32 kB
Mlocked: 32 kB
HighTotal: 2226632 kB
HighFree: 538868 kB
LowTotal: 877852 kB
LowFree: 800448 kB
SwapTotal: 1999540 kB
SwapFree: 1999540 kB
Dirty: 28 kB
Writeback: 0 kB
AnonPages: 81328 kB
Mapped: 44056 kB
Shmem: 1458868 kB
Slab: 26312 kB
SReclaimable: 18636 kB
SUnreclaim: 7676 kB
KernelStack: 1816 kB
PageTables: 2424 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3551780 kB
Committed_AS: 1771932 kB
VmallocTotal: 122880 kB
VmallocUsed: 16308 kB
VmallocChunk: 94312 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 20472 kB
DirectMap4M: 888832 kB

> % cat /proc/zoneinfo

Node 0, zone DMA
pages free 2614
min 16
low 20
high 24
scanned 0
spanned 4080
present 3951
nr_free_pages 2614
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 0
nr_active_file 0
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 0
nr_file_pages 0
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 2
nr_page_table_pages 0
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 865, 3022, 3022)
pagesets
cpu: 0
count: 0
high: 0
batch: 1
vm stats threshold: 4
cpu: 1
count: 0
high: 0
batch: 1
vm stats threshold: 4
all_unreclaimable: 0
prev_priority: 10
start_pfn: 16
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 0
recent_file_scanned: 0
anon_ratio: 30
Node 0, zone Normal
pages free 197498
min 932
low 1165
high 1398
scanned 0
spanned 223230
present 221486
nr_free_pages 197498
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 7317
nr_active_file 3467
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 1
nr_file_pages 10783
nr_dirty 7
nr_writeback 0
nr_slab_reclaimable 4659
nr_slab_unreclaimable 1917
nr_page_table_pages 0
nr_kernel_stack 227
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 0, 17259, 17259)
pagesets
cpu: 0
count: 151
high: 186
batch: 31
vm stats threshold: 16
cpu: 1
count: 80
high: 186
batch: 31
vm stats threshold: 16
all_unreclaimable: 0
prev_priority: 10
start_pfn: 4096
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 1
recent_file_scanned: 4
anon_ratio: 14
Node 0, zone HighMem
pages free 134717
min 128
low 709
high 1290
scanned 0
spanned 556658
present 552309
nr_free_pages 134717
nr_inactive_anon 6387
nr_active_anon 378651
nr_inactive_file 26887
nr_active_file 7181
nr_unevictable 8
nr_mlock 8
nr_anon_pages 20369
nr_mapped 11017
nr_file_pages 398789
nr_dirty 1
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 0
nr_page_table_pages 606
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 364721
protection: (0, 0, 0, 0)
pagesets
cpu: 0
count: 155
high: 186
batch: 31
vm stats threshold: 24
cpu: 1
count: 150
high: 186
batch: 31
vm stats threshold: 24
all_unreclaimable: 0
prev_priority: 10
start_pfn: 227326
inactive_ratio: 4
recent_anon_rotated: 3636
recent_anon_scanned: 11656
recent_file_rotated: 0
recent_file_scanned: 1
anon_ratio: 40

> # echo 1 > /proc/sys/vm/shrink_all_memory
> # tail /var/log/messages

[ 627.410562] shrink_all_memory start
[ 627.410889] shrink_all_memory: req:1 reclaimed:32 free:334798

> % cat /proc/meminfo

MemTotal: 3104484 kB
MemFree: 1339184 kB
Buffers: 43048 kB
Cached: 1595396 kB
SwapCached: 0 kB
Active: 1556584 kB
Inactive: 163144 kB
Active(anon): 1513976 kB
Inactive(anon): 26336 kB
Active(file): 42608 kB
Inactive(file): 136808 kB
Unevictable: 32 kB
Mlocked: 32 kB
HighTotal: 2226632 kB
HighFree: 538744 kB
LowTotal: 877852 kB
LowFree: 800440 kB
SwapTotal: 1999540 kB
SwapFree: 1999540 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 81340 kB
Mapped: 44104 kB
Shmem: 1459032 kB
Slab: 26316 kB
SReclaimable: 18640 kB
SUnreclaim: 7676 kB
KernelStack: 1824 kB
PageTables: 2424 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3551780 kB
Committed_AS: 1772116 kB
VmallocTotal: 122880 kB
VmallocUsed: 16308 kB
VmallocChunk: 94312 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 20472 kB
DirectMap4M: 888832 kB

> % cat /proc/zoneinfo

Node 0, zone DMA
pages free 2614
min 16
low 20
high 24
scanned 0
spanned 4080
present 3951
nr_free_pages 2614
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 0
nr_active_file 0
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 0
nr_file_pages 0
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 2
nr_page_table_pages 0
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 865, 3022, 3022)
pagesets
cpu: 0
count: 0
high: 0
batch: 1
vm stats threshold: 4
cpu: 1
count: 0
high: 0
batch: 1
vm stats threshold: 4
all_unreclaimable: 0
prev_priority: 9
start_pfn: 16
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 0
recent_file_scanned: 0
anon_ratio: 30
Node 0, zone Normal
pages free 197496
min 932
low 1165
high 1398
scanned 0
spanned 223230
present 221486
nr_free_pages 197496
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 7344
nr_active_file 3471
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 1
nr_file_pages 10814
nr_dirty 27
nr_writeback 0
nr_slab_reclaimable 4660
nr_slab_unreclaimable 1917
nr_page_table_pages 0
nr_kernel_stack 228
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 0, 17259, 17259)
pagesets
cpu: 0
count: 116
high: 186
batch: 31
vm stats threshold: 16
cpu: 1
count: 61
high: 186
batch: 31
vm stats threshold: 16
all_unreclaimable: 0
prev_priority: 9
start_pfn: 4096
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 5
recent_file_scanned: 39
anon_ratio: 6
Node 0, zone HighMem
pages free 133632
min 128
low 709
high 1290
scanned 0
spanned 556658
present 552309
nr_free_pages 133632
nr_inactive_anon 7700
nr_active_anon 378501
nr_inactive_file 26857
nr_active_file 7181
nr_unevictable 8
nr_mlock 8
nr_anon_pages 20335
nr_mapped 11025
nr_file_pages 399912
nr_dirty 1
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 0
nr_page_table_pages 606
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 365874
protection: (0, 0, 0, 0)
pagesets
cpu: 0
count: 90
high: 186
batch: 31
vm stats threshold: 24
cpu: 1
count: 164
high: 186
batch: 31
vm stats threshold: 24
all_unreclaimable: 0
prev_priority: 9
start_pfn: 227326
inactive_ratio: 4
recent_anon_rotated: 7752
recent_anon_scanned: 27187
recent_file_rotated: 0
recent_file_scanned: 42
anon_ratio: 3

2010-08-23 00:22:09

by KOSAKI Motohiro

[permalink] [raw]
Subject: Re: [Bisected Regression in 2.6.35] A full tmpfs filesystem causes hibernationto hang


sorry, my last mail has some mistake.

> Hmm...
> I've tested hibernation case for a while. but I have no luck. I couldn't
> reproduce your issue. Very sorry. Can you please help our debugging?
> If possible, I hope to run following three test.
>
> 1. Please let me know your machine & test script
>

% ./your-fills-up-the-tmpfs-script (i.e. run your script)

> % cat /proc/meminfo
> % cat /proc/vmstat
> % cat /proc/zoneinfo
> % df
> % cat your-fills-up-the-tmpfs-script
>
> 2. call shrink_all_memory() forcely and show result
>


% ./your-fills-up-the-tmpfs-script

> % cat /proc/meminfo
> % cat /proc/zoneinfo
> # echo 1 > /proc/sys/vm/shrink_all_memory
> # tail /var/log/messages
> % cat /proc/meminfo
> % cat /proc/zoneinfo
>
>
> 3. reset zone_reclaim_stat and rerun shrink_all_memory
>


% ./your-fills-up-the-tmpfs-script

> # echo 1 > /proc/sys/vm/reset_reclaim_stat
> % cat /proc/meminfo
> % cat /proc/zoneinfo
> # echo 1 > /proc/sys/vm/shrink_all_memory
> # tail /var/log/messages
> % cat /proc/meminfo
> % cat /proc/zoneinfo
>
>


2010-08-23 06:28:01

by M. Vefa Bicakci

[permalink] [raw]
Subject: Re: [Bisected Regression in 2.6.35] A full tmpfs filesystem causeshibernation to hang

Hello,

On 23/08/10 03:22 AM, KOSAKI Motohiro wrote:
>
> sorry, my last mail has some mistake.
>

No problem. I did the tests again according to your new instructions,
and I am appending the results.

Regards,

M. Vefa Bicakci

> 1. Please let me know your machine & test script
>

% ./your-fills-up-the-tmpfs-script (i.e. run your script)

> % cat /proc/meminfo

MemTotal: 3104484 kB
MemFree: 1258780 kB
Buffers: 30820 kB
Cached: 1693836 kB
SwapCached: 0 kB
Active: 1670344 kB
Inactive: 137960 kB
Active(anon): 1633940 kB
Inactive(anon): 26224 kB
Active(file): 36404 kB
Inactive(file): 111736 kB
Unevictable: 32 kB
Mlocked: 32 kB
HighTotal: 2226632 kB
HighFree: 437684 kB
LowTotal: 877852 kB
LowFree: 821096 kB
SwapTotal: 1999540 kB
SwapFree: 1999540 kB
Dirty: 28 kB
Writeback: 0 kB
AnonPages: 83676 kB
Mapped: 44220 kB
Shmem: 1576520 kB
Slab: 17136 kB
SReclaimable: 9480 kB
SUnreclaim: 7656 kB
KernelStack: 1832 kB
PageTables: 2440 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3551780 kB
Committed_AS: 1892636 kB
VmallocTotal: 122880 kB
VmallocUsed: 16308 kB
VmallocChunk: 92764 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 24568 kB
DirectMap4M: 884736 kB


> % cat /proc/vmstat

nr_free_pages 314850
nr_inactive_anon 6480
nr_active_anon 408485
nr_inactive_file 27935
nr_active_file 9101
nr_unevictable 8
nr_mlock 8
nr_anon_pages 20919
nr_mapped 11055
nr_file_pages 431089
nr_dirty 8
nr_writeback 0
nr_slab_reclaimable 2370
nr_slab_unreclaimable 1914
nr_page_table_pages 610
nr_kernel_stack 229
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 394054
pgpgin 150719
pgpgout 5312
pswpin 0
pswpout 0
pgalloc_dma 2
pgalloc_normal 116354
pgalloc_high 2637682
pgalloc_movable 0
pgfree 3069531
pgactivate 2339016
pgdeactivate 160
pgfault 699155
pgmajfault 1082
pgrefill_dma 0
pgrefill_normal 0
pgrefill_high 160
pgrefill_movable 0
pgsteal_dma 0
pgsteal_normal 0
pgsteal_high 29
pgsteal_movable 0
pgscan_kswapd_dma 0
pgscan_kswapd_normal 0
pgscan_kswapd_high 0
pgscan_kswapd_movable 0
pgscan_direct_dma 0
pgscan_direct_normal 0
pgscan_direct_high 32
pgscan_direct_movable 0
pginodesteal 0
slabs_scanned 0
kswapd_steal 0
kswapd_inodesteal 0
kswapd_low_wmark_hit_quickly 0
kswapd_high_wmark_hit_quickly 0
kswapd_skip_congestion_wait 0
pageoutrun 0
allocstall 1
pgrotated 0
htlb_buddy_alloc_success 0
htlb_buddy_alloc_fail 0
unevictable_pgs_culled 7832
unevictable_pgs_scanned 0
unevictable_pgs_rescued 7046
unevictable_pgs_mlocked 8562
unevictable_pgs_munlocked 8554
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
unevictable_pgs_mlockfreed 0

> % cat /proc/zoneinfo

Node 0, zone DMA
pages free 2614
min 16
low 20
high 24
scanned 0
spanned 4080
present 3951
nr_free_pages 2614
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 0
nr_active_file 0
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 0
nr_file_pages 0
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 2
nr_page_table_pages 0
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 865, 3022, 3022)
pagesets
cpu: 0
count: 0
high: 0
batch: 1
vm stats threshold: 4
cpu: 1
count: 0
high: 0
batch: 1
vm stats threshold: 4
all_unreclaimable: 0
prev_priority: 9
start_pfn: 16
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 0
recent_file_scanned: 0
anon_ratio: 30
Node 0, zone Normal
pages free 202722
min 932
low 1165
high 1398
scanned 0
spanned 223230
present 221486
nr_free_pages 202722
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 5792
nr_active_file 1970
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 1
nr_file_pages 7761
nr_dirty 7
nr_writeback 0
nr_slab_reclaimable 2370
nr_slab_unreclaimable 1912
nr_page_table_pages 0
nr_kernel_stack 229
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 0, 17259, 17259)
pagesets
cpu: 0
count: 155
high: 186
batch: 31
vm stats threshold: 16
cpu: 1
count: 173
high: 186
batch: 31
vm stats threshold: 16
all_unreclaimable: 0
prev_priority: 9
start_pfn: 4096
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 272
recent_file_scanned: 1385
anon_ratio: 7
Node 0, zone HighMem
pages free 109607
min 128
low 709
high 1290
scanned 0
spanned 556658
present 552309
nr_free_pages 109607
nr_inactive_anon 6482
nr_active_anon 408478
nr_inactive_file 22146
nr_active_file 7131
nr_unevictable 8
nr_mlock 8
nr_anon_pages 20919
nr_mapped 11054
nr_file_pages 423333
nr_dirty 2
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 0
nr_page_table_pages 610
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 394056
protection: (0, 0, 0, 0)
pagesets
cpu: 0
count: 107
high: 186
batch: 31
vm stats threshold: 24
cpu: 1
count: 129
high: 186
batch: 31
vm stats threshold: 24
all_unreclaimable: 0
prev_priority: 9
start_pfn: 227326
inactive_ratio: 4
recent_anon_rotated: 1259857
recent_anon_scanned: 2522124
recent_file_rotated: 1269
recent_file_scanned: 5591
anon_ratio: 16

> % df

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda15 10317828 7309436 2484276 75% /
tmpfs 1552240 8 1552232 1% /lib/init/rw
udev 1549148 220 1548928 1% /dev
tmpfs 1552240 4 1552236 1% /dev/shm
tmpfs 1552240 1552240 0 100% /tmp

> % cat your-fills-up-the-tmpfs-script

#!/bin/bash

rm -vf /tmp/tmp
dd if=/dev/zero of=/tmp/tmp bs=1024k count=1600
shred -vn1 /tmp/tmp

exit 0 ### removed when testing hibernation

read -p "Press enter to hibernate..."

echo -n reboot > /sys/power/disk
echo -n disk > /sys/power/state

echo "Exit code: $?"

rm -vf /tmp/tmp


>
> 2. call shrink_all_memory() forcely and show result
>

% ./your-fills-up-the-tmpfs-script

> % cat /proc/meminfo

MemTotal: 3104484 kB
MemFree: 1259400 kB
Buffers: 30836 kB
Cached: 1693572 kB
SwapCached: 0 kB
Active: 1670356 kB
Inactive: 137708 kB
Active(anon): 1633948 kB
Inactive(anon): 25944 kB
Active(file): 36408 kB
Inactive(file): 111764 kB
Unevictable: 32 kB
Mlocked: 32 kB
HighTotal: 2226632 kB
HighFree: 438056 kB
LowTotal: 877852 kB
LowFree: 821344 kB
SwapTotal: 1999540 kB
SwapFree: 1999540 kB
Dirty: 36 kB
Writeback: 0 kB
AnonPages: 83684 kB
Mapped: 44236 kB
Shmem: 1576240 kB
Slab: 17136 kB
SReclaimable: 9480 kB
SUnreclaim: 7656 kB
KernelStack: 1832 kB
PageTables: 2440 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3551780 kB
Committed_AS: 1892412 kB
VmallocTotal: 122880 kB
VmallocUsed: 16308 kB
VmallocChunk: 92764 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 24568 kB
DirectMap4M: 884736 kB

> % cat /proc/zoneinfo

Node 0, zone DMA
pages free 2614
min 16
low 20
high 24
scanned 0
spanned 4080
present 3951
nr_free_pages 2614
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 0
nr_active_file 0
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 0
nr_file_pages 0
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 2
nr_page_table_pages 0
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 865, 3022, 3022)
pagesets
cpu: 0
count: 0
high: 0
batch: 1
vm stats threshold: 4
cpu: 1
count: 0
high: 0
batch: 1
vm stats threshold: 4
all_unreclaimable: 0
prev_priority: 9
start_pfn: 16
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 0
recent_file_scanned: 0
anon_ratio: 30
Node 0, zone Normal
pages free 202691
min 932
low 1165
high 1398
scanned 0
spanned 223230
present 221486
nr_free_pages 202691
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 5796
nr_active_file 1970
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 1
nr_file_pages 7765
nr_dirty 7
nr_writeback 0
nr_slab_reclaimable 2370
nr_slab_unreclaimable 1912
nr_page_table_pages 0
nr_kernel_stack 229
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 0, 17259, 17259)
pagesets
cpu: 0
count: 161
high: 186
batch: 31
vm stats threshold: 16
cpu: 1
count: 170
high: 186
batch: 31
vm stats threshold: 16
all_unreclaimable: 0
prev_priority: 9
start_pfn: 4096
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 272
recent_file_scanned: 1389
anon_ratio: 7
Node 0, zone HighMem
pages free 109514
min 128
low 709
high 1290
scanned 0
spanned 556658
present 552309
nr_free_pages 109514
nr_inactive_anon 6484
nr_active_anon 408487
nr_inactive_file 22148
nr_active_file 7132
nr_unevictable 8
nr_mlock 8
nr_anon_pages 20921
nr_mapped 11058
nr_file_pages 423338
nr_dirty 3
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 0
nr_page_table_pages 610
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 394058
protection: (0, 0, 0, 0)
pagesets
cpu: 0
count: 167
high: 186
batch: 31
vm stats threshold: 24
cpu: 1
count: 156
high: 186
batch: 31
vm stats threshold: 24
all_unreclaimable: 0
prev_priority: 9
start_pfn: 227326
inactive_ratio: 4
recent_anon_rotated: 1650360
recent_anon_scanned: 3304957
recent_file_rotated: 1270
recent_file_scanned: 5595
anon_ratio: 16

> # echo 1 > /proc/sys/vm/shrink_all_memory

> # tail /var/log/messages

[ 439.165652] shrink_all_memory start
[ 439.165960] shrink_all_memory: req:1 reclaimed:32 free:313765

> % cat /proc/meminfo

MemTotal: 3104484 kB
MemFree: 1259516 kB
Buffers: 30852 kB
Cached: 1693672 kB
SwapCached: 0 kB
Active: 1669728 kB
Inactive: 138456 kB
Active(anon): 1633320 kB
Inactive(anon): 26800 kB
Active(file): 36408 kB
Inactive(file): 111656 kB
Unevictable: 32 kB
Mlocked: 32 kB
HighTotal: 2226632 kB
HighFree: 438180 kB
LowTotal: 877852 kB
LowFree: 821336 kB
SwapTotal: 1999540 kB
SwapFree: 1999540 kB
Dirty: 40 kB
Writeback: 0 kB
AnonPages: 83696 kB
Mapped: 44236 kB
Shmem: 1576460 kB
Slab: 17140 kB
SReclaimable: 9484 kB
SUnreclaim: 7656 kB
KernelStack: 1840 kB
PageTables: 2440 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3551780 kB
Committed_AS: 1892540 kB
VmallocTotal: 122880 kB
VmallocUsed: 16308 kB
VmallocChunk: 92764 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 24568 kB
DirectMap4M: 884736 kB

> % cat /proc/zoneinfo

Node 0, zone DMA
pages free 2614
min 16
low 20
high 24
scanned 0
spanned 4080
present 3951
nr_free_pages 2614
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 0
nr_active_file 0
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 0
nr_file_pages 0
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 2
nr_page_table_pages 0
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 865, 3022, 3022)
pagesets
cpu: 0
count: 0
high: 0
batch: 1
vm stats threshold: 4
cpu: 1
count: 0
high: 0
batch: 1
vm stats threshold: 4
all_unreclaimable: 0
prev_priority: 9
start_pfn: 16
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 0
recent_file_scanned: 0
anon_ratio: 30
Node 0, zone Normal
pages free 202720
min 932
low 1165
high 1398
scanned 0
spanned 223230
present 221486
nr_free_pages 202720
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 5798
nr_active_file 1970
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 1
nr_file_pages 7767
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 2371
nr_slab_unreclaimable 1912
nr_page_table_pages 0
nr_kernel_stack 230
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 0, 17259, 17259)
pagesets
cpu: 0
count: 139
high: 186
batch: 31
vm stats threshold: 16
cpu: 1
count: 165
high: 186
batch: 31
vm stats threshold: 16
all_unreclaimable: 0
prev_priority: 9
start_pfn: 4096
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 272
recent_file_scanned: 1391
anon_ratio: 7
Node 0, zone HighMem
pages free 109576
min 128
low 709
high 1290
scanned 0
spanned 556658
present 552309
nr_free_pages 109576
nr_inactive_anon 6642
nr_active_anon 408330
nr_inactive_file 22118
nr_active_file 7132
nr_unevictable 8
nr_mlock 8
nr_anon_pages 20924
nr_mapped 11058
nr_file_pages 423306
nr_dirty 3
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 0
nr_page_table_pages 610
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 394056
protection: (0, 0, 0, 0)
pagesets
cpu: 0
count: 158
high: 186
batch: 31
vm stats threshold: 24
cpu: 1
count: 133
high: 186
batch: 31
vm stats threshold: 24
all_unreclaimable: 0
prev_priority: 9
start_pfn: 227326
inactive_ratio: 4
recent_anon_rotated: 103467
recent_anon_scanned: 207085
recent_file_rotated: 1270
recent_file_scanned: 5629
anon_ratio: 16

> 3. reset zone_reclaim_stat and rerun shrink_all_memory
>

% ./your-fills-up-the-tmpfs-script

> # echo 1 > /proc/sys/vm/reset_reclaim_stat
> % cat /proc/meminfo

MemTotal: 3104484 kB
MemFree: 1260012 kB
Buffers: 30884 kB
Cached: 1693496 kB
SwapCached: 0 kB
Active: 1669736 kB
Inactive: 138316 kB
Active(anon): 1633324 kB
Inactive(anon): 26616 kB
Active(file): 36412 kB
Inactive(file): 111700 kB
Unevictable: 32 kB
Mlocked: 32 kB
HighTotal: 2226632 kB
HighFree: 438304 kB
LowTotal: 877852 kB
LowFree: 821708 kB
SwapTotal: 1999540 kB
SwapFree: 1999540 kB
Dirty: 12 kB
Writeback: 0 kB
AnonPages: 83700 kB
Mapped: 44240 kB
Shmem: 1576272 kB
Slab: 17140 kB
SReclaimable: 9484 kB
SUnreclaim: 7656 kB
KernelStack: 1840 kB
PageTables: 2440 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3551780 kB
Committed_AS: 1892440 kB
VmallocTotal: 122880 kB
VmallocUsed: 16308 kB
VmallocChunk: 92764 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 24568 kB
DirectMap4M: 884736 kB

> % cat /proc/zoneinfo


Node 0, zone DMA
pages free 2614
min 16
low 20
high 24
scanned 0
spanned 4080
present 3951
nr_free_pages 2614
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 0
nr_active_file 0
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 0
nr_file_pages 0
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 2
nr_page_table_pages 0
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 865, 3022, 3022)
pagesets
cpu: 0
count: 0
high: 0
batch: 1
vm stats threshold: 4
cpu: 1
count: 0
high: 0
batch: 1
vm stats threshold: 4
all_unreclaimable: 0
prev_priority: 9
start_pfn: 16
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 0
recent_file_scanned: 0
anon_ratio: 30
Node 0, zone Normal
pages free 202844
min 932
low 1165
high 1398
scanned 0
spanned 223230
present 221486
nr_free_pages 202844
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 5808
nr_active_file 1970
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 1
nr_file_pages 7777
nr_dirty 5
nr_writeback 0
nr_slab_reclaimable 2371
nr_slab_unreclaimable 1912
nr_page_table_pages 0
nr_kernel_stack 230
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 0, 17259, 17259)
pagesets
cpu: 0
count: 153
high: 186
batch: 31
vm stats threshold: 16
cpu: 1
count: 39
high: 186
batch: 31
vm stats threshold: 16
all_unreclaimable: 0
prev_priority: 9
start_pfn: 4096
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 0
recent_file_scanned: 2
anon_ratio: 12
Node 0, zone HighMem
pages free 109607
min 128
low 709
high 1290
scanned 0
spanned 556658
present 552309
nr_free_pages 109607
nr_inactive_anon 6638
nr_active_anon 408331
nr_inactive_file 22120
nr_active_file 7133
nr_unevictable 8
nr_mlock 8
nr_anon_pages 20925
nr_mapped 11057
nr_file_pages 423305
nr_dirty 1
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 0
nr_page_table_pages 610
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 394052
protection: (0, 0, 0, 0)
pagesets
cpu: 0
count: 157
high: 186
batch: 31
vm stats threshold: 24
cpu: 1
count: 94
high: 186
batch: 31
vm stats threshold: 24
all_unreclaimable: 0
prev_priority: 9
start_pfn: 227326
inactive_ratio: 4
recent_anon_rotated: 896
recent_anon_scanned: 1078
recent_file_rotated: 0
recent_file_scanned: 1
anon_ratio: 20

> # echo 1 > /proc/sys/vm/shrink_all_memory
> # tail /var/log/messages

[ 504.609297] shrink_all_memory start
[ 504.609537] shrink_all_memory: req:1 reclaimed:31 free:315005

> % cat /proc/meminfo


MemTotal: 3104484 kB
MemFree: 1255796 kB
Buffers: 30788 kB
Cached: 1697944 kB
SwapCached: 0 kB
Active: 1669480 kB
Inactive: 143044 kB
Active(anon): 1633064 kB
Inactive(anon): 31432 kB
Active(file): 36416 kB
Inactive(file): 111612 kB
Unevictable: 32 kB
Mlocked: 32 kB
HighTotal: 2226632 kB
HighFree: 433964 kB
LowTotal: 877852 kB
LowFree: 821832 kB
SwapTotal: 1999540 kB
SwapFree: 1999540 kB
Dirty: 40 kB
Writeback: 0 kB
AnonPages: 83852 kB
Mapped: 44220 kB
Shmem: 1580704 kB
Slab: 17140 kB
SReclaimable: 9484 kB
SUnreclaim: 7656 kB
KernelStack: 1840 kB
PageTables: 2440 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3551780 kB
Committed_AS: 1896832 kB
VmallocTotal: 122880 kB
VmallocUsed: 16308 kB
VmallocChunk: 92764 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 24568 kB
DirectMap4M: 884736 kB

> % cat /proc/zoneinfo

Node 0, zone DMA
pages free 2614
min 16
low 20
high 24
scanned 0
spanned 4080
present 3951
nr_free_pages 2614
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 0
nr_active_file 0
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 0
nr_file_pages 0
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 2
nr_page_table_pages 0
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 865, 3022, 3022)
pagesets
cpu: 0
count: 0
high: 0
batch: 1
vm stats threshold: 4
cpu: 1
count: 0
high: 0
batch: 1
vm stats threshold: 4
all_unreclaimable: 0
prev_priority: 10
start_pfn: 16
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 0
recent_file_scanned: 0
anon_ratio: 30
Node 0, zone Normal
pages free 202844
min 932
low 1165
high 1398
scanned 0
spanned 223230
present 221486
nr_free_pages 202844
nr_inactive_anon 0
nr_active_anon 0
nr_inactive_file 5778
nr_active_file 1971
nr_unevictable 0
nr_mlock 0
nr_anon_pages 0
nr_mapped 1
nr_file_pages 7749
nr_dirty 5
nr_writeback 0
nr_slab_reclaimable 2371
nr_slab_unreclaimable 1912
nr_page_table_pages 0
nr_kernel_stack 230
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
protection: (0, 0, 17259, 17259)
pagesets
cpu: 0
count: 102
high: 186
batch: 31
vm stats threshold: 16
cpu: 1
count: 63
high: 186
batch: 31
vm stats threshold: 16
all_unreclaimable: 0
prev_priority: 10
start_pfn: 4096
inactive_ratio: 1
recent_anon_rotated: 0
recent_anon_scanned: 0
recent_file_rotated: 1
recent_file_scanned: 38
anon_ratio: 2
Node 0, zone HighMem
pages free 109545
min 128
low 709
high 1290
scanned 96
spanned 556658
present 552309
nr_free_pages 109545
nr_inactive_anon 6797
nr_active_anon 408235
nr_inactive_file 22122
nr_active_file 7133
nr_unevictable 8
nr_mlock 8
nr_anon_pages 20925
nr_mapped 11054
nr_file_pages 423370
nr_dirty 5
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 0
nr_page_table_pages 610
nr_kernel_stack 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 394115
protection: (0, 0, 0, 0)
pagesets
cpu: 0
count: 137
high: 186
batch: 31
vm stats threshold: 24
cpu: 1
count: 119
high: 186
batch: 31
vm stats threshold: 24
all_unreclaimable: 0
prev_priority: 10
start_pfn: 227326
inactive_ratio: 4
recent_anon_rotated: 1358
recent_anon_scanned: 1730
recent_file_rotated: 0
recent_file_scanned: 3
anon_ratio: 11

2010-08-25 00:48:10

by KOSAKI Motohiro

[permalink] [raw]
Subject: Re: [Bisected Regression in 2.6.35] A full tmpfs filesystem causeshibernation to hang

> Hello,
>
> On 23/08/10 03:22 AM, KOSAKI Motohiro wrote:
> >
> > sorry, my last mail has some mistake.
> >
>
> No problem. I did the tests again according to your new instructions,
> and I am appending the results.

Great!

> > 1. Please let me know your machine & test script
> >
>
> % ./your-fills-up-the-tmpfs-script (i.e. run your script)
>
> > % cat /proc/meminfo
>
> MemTotal: 3104484 kB
> MemFree: 1258780 kB

Hm, free memory is enough lots.


> Buffers: 30820 kB
> Cached: 1693836 kB
> SwapCached: 0 kB
> Active: 1670344 kB
> Inactive: 137960 kB
> Active(anon): 1633940 kB
> Inactive(anon): 26224 kB
> Active(file): 36404 kB
> Inactive(file): 111736 kB
> Unevictable: 32 kB
> Mlocked: 32 kB
> HighTotal: 2226632 kB
> HighFree: 437684 kB

You are using 32 bit system.

> LowTotal: 877852 kB
> LowFree: 821096 kB

low memory is empty!


> SwapTotal: 1999540 kB
> SwapFree: 1999540 kB
> Dirty: 28 kB
> Writeback: 0 kB
> AnonPages: 83676 kB
> Mapped: 44220 kB
> Shmem: 1576520 kB

tmpfs files are using about 1.5GB memory.
(because tmpfs is limited totalmem/2 by default)


> Slab: 17136 kB
> SReclaimable: 9480 kB
> SUnreclaim: 7656 kB
> KernelStack: 1832 kB
> PageTables: 2440 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 3551780 kB
> Committed_AS: 1892636 kB
> VmallocTotal: 122880 kB
> VmallocUsed: 16308 kB
> VmallocChunk: 92764 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 4096 kB
> DirectMap4k: 24568 kB
> DirectMap4M: 884736 kB


> > % cat /proc/zoneinfo
>
> Node 0, zone DMA
> recent_anon_rotated: 0
> recent_anon_scanned: 0
> recent_file_rotated: 0
> recent_file_scanned: 0
> anon_ratio: 30
> Node 0, zone Normal
> recent_anon_rotated: 0
> recent_anon_scanned: 0
> recent_file_rotated: 272
> recent_file_scanned: 1385
> anon_ratio: 7
> Node 0, zone HighMem
> recent_anon_rotated: 1259857
> recent_anon_scanned: 2522124
> recent_file_rotated: 1269
> recent_file_scanned: 5591
> anon_ratio: 16

anon/file reclaim scanning ratio is NOT strange.


>
> > % df
>
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/sda15 10317828 7309436 2484276 75% /
> tmpfs 1552240 8 1552232 1% /lib/init/rw
> udev 1549148 220 1548928 1% /dev
> tmpfs 1552240 4 1552236 1% /dev/shm
> tmpfs 1552240 1552240 0 100% /tmp

Seems typical one drive system. no strange.


> > % cat your-fills-up-the-tmpfs-script
>
> #!/bin/bash
>
> rm -vf /tmp/tmp
> dd if=/dev/zero of=/tmp/tmp bs=1024k count=1600
> shred -vn1 /tmp/tmp
>
> exit 0 ### removed when testing hibernation
>
> read -p "Press enter to hibernate..."
>
> echo -n reboot > /sys/power/disk
> echo -n disk > /sys/power/state
>
> echo "Exit code: $?"
>
> rm -vf /tmp/tmp

Almostly same with my test program ;)


> > # echo 1 > /proc/sys/vm/shrink_all_memory
>
> > # tail /var/log/messages
>
> [ 439.165652] shrink_all_memory start
> [ 439.165960] shrink_all_memory: req:1 reclaimed:32 free:313765
>

My code doesn't makes hang!


Hmm... Hmm...
To be honest, I have no idea why your hang was happen.

1) zone normal is not used. your system don't need additional reclaim
at all.
2) reclaim logic seems to doesn't makes hang.


Can you please try following additional test?

# echo 8 > /proc/sysrq-trigger
# echo disk > /sys/power/state



2010-08-25 08:39:23

by KOSAKI Motohiro

[permalink] [raw]
Subject: Re: [Bisected Regression in 2.6.35] A full tmpfs filesystem causeshibernation to hang

> > #!/bin/bash
> >
> > rm -vf /tmp/tmp
> > dd if=/dev/zero of=/tmp/tmp bs=1024k count=1600
> > shred -vn1 /tmp/tmp
> >
> > exit 0 ### removed when testing hibernation
> >
> > read -p "Press enter to hibernate..."
> >
> > echo -n reboot > /sys/power/disk
> > echo -n disk > /sys/power/state
> >
> > echo "Exit code: $?"
> >
> > rm -vf /tmp/tmp
>
> Almostly same with my test program ;)

Can you please try to avoid to use /tmp. As I said,

mount -t tmpfs none /mnt/another_tmpfs
dd if=/dev/zero of=/mnt/another_tmpfs/tmp bs=1024k count=1600
shred -vn1 /mnt/another_tmpfs/tmp


That said, We need to know you issue is lots-anon-pages issue or
filesystem-full issue.

2010-08-25 08:55:59

by KOSAKI Motohiro

[permalink] [raw]
Subject: Re: [Bisected Regression in 2.6.35] A full tmpfs filesystem causeshibernation to hang

> > % cat /proc/meminfo
>
> MemTotal: 3104484 kB
> MemFree: 2817616 kB
> Buffers: 31156 kB
> Cached: 142124 kB
> SwapCached: 0 kB
> Active: 116464 kB
> Inactive: 137424 kB
> Active(anon): 80852 kB
> Inactive(anon): 24820 kB
> Active(file): 35612 kB
> Inactive(file): 112604 kB
> Unevictable: 32 kB
> Mlocked: 32 kB
> HighTotal: 2226632 kB
> HighFree: 1994008 kB
> LowTotal: 877852 kB
> LowFree: 823608 kB
> SwapTotal: 1999540 kB
> SwapFree: 1999540 kB

!! Your swap partition is smaller than physical memory. As far as I know,
swap partition need physcal-mem x 2 size. Can you please try to change
swap configuration?

Rafael, please correct me if I'm talking wrong.

Thanks.


2010-08-25 10:11:01

by M. Vefa Bicakci

[permalink] [raw]
Subject: Re: [Bisected Regression in 2.6.35] A full tmpfs filesystem causeshibernationto hang

(Sorry, I am sending this again as I forgot to include LKML in the CC.)

On 25/08/10 11:39 AM, KOSAKI Motohiro wrote:
>
> Can you please try to avoid to use /tmp. As I said,
>
> mount -t tmpfs none /mnt/another_tmpfs
> dd if=/dev/zero of=/mnt/another_tmpfs/tmp bs=1024k count=1600
> shred -vn1 /mnt/another_tmpfs/tmp
>
>
> That said, We need to know you issue is lots-anon-pages issue or
> filesystem-full issue.

Hello,

I have changed my configuration so that I have a 8 gigabyte large
swap partition, and I have also made the changes you suggested
to my hibernation/fill-tmpfs script so that a tmpfs other than
/tmp is used.

Unfortunately, nothing changed. I still get hangs after a few lines
are printed to the console. The last two lines are from your patch.
Here is an observation from an actual test which ended with a hang:

=== 8< ===
PM: Preallocating image memory
shrink_all_memory: start
shrink_all_memory: req: 375019 reclaimed: 48055 free: 326810
=== >8 ===

One thing I should note is that, before your commit, I never had
any problems even though my swap size was not two times the size
of my physical memory and even though I used tmpfs as /tmp.

If there is anything I can do to debug this problem please let me
know.

Regards,

M. Vefa Bicakci

2010-08-25 10:11:45

by M. Vefa Bicakci

[permalink] [raw]
Subject: Re: [Bisected Regression in 2.6.35] A full tmpfs filesystem causeshibernationto hang

On 25/08/10 11:55 AM, KOSAKI Motohiro wrote:
>>> % cat /proc/meminfo
>>
>> MemTotal: 3104484 kB
>> MemFree: 2817616 kB
>> Buffers: 31156 kB
>> Cached: 142124 kB
>> SwapCached: 0 kB
>> Active: 116464 kB
>> Inactive: 137424 kB
>> Active(anon): 80852 kB
>> Inactive(anon): 24820 kB
>> Active(file): 35612 kB
>> Inactive(file): 112604 kB
>> Unevictable: 32 kB
>> Mlocked: 32 kB
>> HighTotal: 2226632 kB
>> HighFree: 1994008 kB
>> LowTotal: 877852 kB
>> LowFree: 823608 kB
>> SwapTotal: 1999540 kB
>> SwapFree: 1999540 kB
>
> !! Your swap partition is smaller than physical memory. As far as I know,
> swap partition need physcal-mem x 2 size. Can you please try to change
> swap configuration?
>
> Rafael, please correct me if I'm talking wrong.
>
> Thanks.

[Please see my reply in the other thread.]

2010-08-25 17:33:14

by Rafael J. Wysocki

[permalink] [raw]
Subject: Re: [Bisected Regression in 2.6.35] A full tmpfs filesystem causeshibernation to hang

On Wednesday, August 25, 2010, KOSAKI Motohiro wrote:
> > > % cat /proc/meminfo
> >
> > MemTotal: 3104484 kB
> > MemFree: 2817616 kB
> > Buffers: 31156 kB
> > Cached: 142124 kB
> > SwapCached: 0 kB
> > Active: 116464 kB
> > Inactive: 137424 kB
> > Active(anon): 80852 kB
> > Inactive(anon): 24820 kB
> > Active(file): 35612 kB
> > Inactive(file): 112604 kB
> > Unevictable: 32 kB
> > Mlocked: 32 kB
> > HighTotal: 2226632 kB
> > HighFree: 1994008 kB
> > LowTotal: 877852 kB
> > LowFree: 823608 kB
> > SwapTotal: 1999540 kB
> > SwapFree: 1999540 kB
>
> !! Your swap partition is smaller than physical memory. As far as I know,
> swap partition need physcal-mem x 2 size. Can you please try to change
> swap configuration?
>
> Rafael, please correct me if I'm talking wrong.

No, we only save 50% of RAM (at most) during hibernation.

Thanks,
Rafael