Hi,
I've been suffering from s2disk hangs again. This time, the hangs
were always before the hibernation image was written out.
They're still frustratingly random. I just started trying to work out
whether doubling PAGES_FOR_IO makes them go away, but they went away
on their own again.
I did manage to capture a backtrace with debug info though. Here it
is for 2.6.33-rc2. (It has also happened on rc1). I was able to get
the line numbers (using gdb, e.g. "info line
*stop_machine_create+0x27"), having built the kernel with debug info.
[top of trace lost due to screen height]
? sync_page (filemap.c:183)
? wait_on_page_bit (filemap.c:506)
? wake_bit_function (wait.c:174)
? shrink_page_list (vmscan.c:696)
? __delayacct_blkio_end (delayacct.c:94)
? finish_wait (list.h:142)
? congestion_wait (backing-dev.c:761)
? shrink_inactive_list (vmscan.c:1193)
? scsi_request_fn (spinlock.h:306)
? blk_run_queue (blk-core.c:434)
? shrink_zone (vmscan.c:1484)
? do_try_to_free_pages (vmscan.c:1684)
? try_to_free_pages (vmscan.c:1848)
? isolate_pages_global (vmscan.c:980)
? __alloc_pages_nodemask (page_alloc.c:1702)
? __get_free_pages (page_alloc.c:1990)
? copy_process (fork.c:237)
? do_fork (fork.c:1443)
? rb_erase
? __switch_to
? kthread
? kernel_thread
? kthread
? kernel_thread_helper
? kthreadd
? kthreadd
? kernel_thread_helper
INFO: task s2disk:2174 blocked for more than 120 seconds
...
Call Trace:
? __switch_to
? schedule_timeout
? check_preempt_wakeup
? wait_for_common
? default_wake_function
? kthread_create:133
? worker_thread
? schedule
? create_workqueue_thread
? worker_thread
? __create_workqueue_key (workqueue.c:1006)
? stop_machine_create (stop_machine.c:121)
? disable_nonboot_cpus (cpu.c:370)
? hibernation_snapshot
? snapshot_ioctl
...
? sys_ioctl
Thanks for everything
Alan
On Saturday 02 January 2010, Alan Jenkins wrote:
> Hi,
Hi,
> I've been suffering from s2disk hangs again. This time, the hangs
> were always before the hibernation image was written out.
>
> They're still frustratingly random. I just started trying to work out
> whether doubling PAGES_FOR_IO makes them go away, but they went away
> on their own again.
>
> I did manage to capture a backtrace with debug info though. Here it
> is for 2.6.33-rc2. (It has also happened on rc1). I was able to get
> the line numbers (using gdb, e.g. "info line
> *stop_machine_create+0x27"), having built the kernel with debug info.
>
> [top of trace lost due to screen height]
> ? sync_page (filemap.c:183)
> ? wait_on_page_bit (filemap.c:506)
> ? wake_bit_function (wait.c:174)
> ? shrink_page_list (vmscan.c:696)
> ? __delayacct_blkio_end (delayacct.c:94)
> ? finish_wait (list.h:142)
> ? congestion_wait (backing-dev.c:761)
> ? shrink_inactive_list (vmscan.c:1193)
> ? scsi_request_fn (spinlock.h:306)
> ? blk_run_queue (blk-core.c:434)
> ? shrink_zone (vmscan.c:1484)
> ? do_try_to_free_pages (vmscan.c:1684)
> ? try_to_free_pages (vmscan.c:1848)
> ? isolate_pages_global (vmscan.c:980)
> ? __alloc_pages_nodemask (page_alloc.c:1702)
> ? __get_free_pages (page_alloc.c:1990)
> ? copy_process (fork.c:237)
> ? do_fork (fork.c:1443)
> ? rb_erase
> ? __switch_to
> ? kthread
> ? kernel_thread
> ? kthread
> ? kernel_thread_helper
> ? kthreadd
> ? kthreadd
> ? kernel_thread_helper
>
> INFO: task s2disk:2174 blocked for more than 120 seconds
This looks like we have run out of memory while creating a new kernel thread
and we have blocked on I/O while trying to free some space (quite obviously,
because the I/O doesn't work at this point).
I think it should help if you increase PAGES_FOR_IO, then.
Rafael
On 1/2/10, Rafael J. Wysocki <[email protected]> wrote:
> On Saturday 02 January 2010, Alan Jenkins wrote:
> Hi,
>
>> I've been suffering from s2disk hangs again. This time, the hangs
>> were always before the hibernation image was written out.
>>
>> They're still frustratingly random. I just started trying to work out
>> whether doubling PAGES_FOR_IO makes them go away, but they went away
>> on their own again.
>>
>> I did manage to capture a backtrace with debug info though. Here it
>> is for 2.6.33-rc2. (It has also happened on rc1). I was able to get
>> the line numbers (using gdb, e.g. "info line
>> *stop_machine_create+0x27"), having built the kernel with debug info.
>>
>> [top of trace lost due to screen height]
>> ? sync_page (filemap.c:183)
>> ? wait_on_page_bit (filemap.c:506)
>> ? wake_bit_function (wait.c:174)
>> ? shrink_page_list (vmscan.c:696)
>> ? __delayacct_blkio_end (delayacct.c:94)
>> ? finish_wait (list.h:142)
>> ? congestion_wait (backing-dev.c:761)
>> ? shrink_inactive_list (vmscan.c:1193)
>> ? scsi_request_fn (spinlock.h:306)
>> ? blk_run_queue (blk-core.c:434)
>> ? shrink_zone (vmscan.c:1484)
>> ? do_try_to_free_pages (vmscan.c:1684)
>> ? try_to_free_pages (vmscan.c:1848)
>> ? isolate_pages_global (vmscan.c:980)
>> ? __alloc_pages_nodemask (page_alloc.c:1702)
>> ? __get_free_pages (page_alloc.c:1990)
>> ? copy_process (fork.c:237)
>> ? do_fork (fork.c:1443)
>> ? rb_erase
>> ? __switch_to
>> ? kthread
>> ? kernel_thread
>> ? kthread
>> ? kernel_thread_helper
>> ? kthreadd
>> ? kthreadd
>> ? kernel_thread_helper
>>
>> INFO: task s2disk:2174 blocked for more than 120 seconds
>
> This looks like we have run out of memory while creating a new kernel thread
> and we have blocked on I/O while trying to free some space (quite obviously,
> because the I/O doesn't work at this point).
For context, the kernel thread being created here is the stop_machine
thread. It is created by disable_nonboot_cpus(), called from
hibernation_snapshot(). See e.g. this hung task backtrace -
http://picasaweb.google.com/lh/photo/BkKUwZCrQ2ceBIM9ZOh7Ow?feat=directlink
> I think it should help if you increase PAGES_FOR_IO, then.
Ok, it's been happening again on 2.6.33-rc6. Unfortunately increasing
PAGES_FOR_IO doesn't help.
I've been using a test patch to make PAGES_FOR_IO tunable at run time.
I get the same hang if I increase it by a factor of 10, to 10240:
# cd /sys/module/kernel/parameters/
# ls
consoleblank initcall_debug PAGES_FOR_IO panic pause_on_oops SPARE_PAGES
# echo 10240 > PAGES_FOR_IO
# echo 2560 > SPARE_PAGES
# cat SPARE_PAGES
2560
# cat PAGES_FOR_IO
10240
I also added a debug patch to try and understand the calculations with
PAGES_FOR_IO in hibernate_preallocate_memory(). I still don't really
understand them and there could easily be errors in my debug patch,
but the output is interesting.
Increasing PAGES_FOR_IO by almost 10000 has the expected effect of
decreasing "max_size" by the same amount. However it doesn't appear
to increase the number of free pages at the critical moment.
PAGES_FOR_IO = 1024:
http://picasaweb.google.com/lh/photo/DYQGvB_4hvCvVuxZf2ibxg?feat=directlink
PAGES_FOR_IO = 10240:
http://picasaweb.google.com/lh/photo/AIkV_ZBwt22nzN-JdOJCWA?feat=directlink
You may remember that I was originally able to avoid the hang by
reverting commit 5f8dcc2. It doesn't revert cleanly any more.
However, I tried applying my test&debug patches on top of 5f8dcc2~1
(just before the commit that triggered the hang). That kernel
apparently left ~5000 pages free at hibernation time, v.s. ~1200 when
testing the same scenario on 2.6.33-rc6. (As before, the number of
free pages remained the same if I increased PAGES_FOR_IO to 10240).
Regards
Alan
On Tuesday 02 February 2010, Alan Jenkins wrote:
> On 1/2/10, Rafael J. Wysocki <[email protected]> wrote:
> > On Saturday 02 January 2010, Alan Jenkins wrote:
> > Hi,
> >
> >> I've been suffering from s2disk hangs again. This time, the hangs
> >> were always before the hibernation image was written out.
> >>
> >> They're still frustratingly random. I just started trying to work out
> >> whether doubling PAGES_FOR_IO makes them go away, but they went away
> >> on their own again.
> >>
> >> I did manage to capture a backtrace with debug info though. Here it
> >> is for 2.6.33-rc2. (It has also happened on rc1). I was able to get
> >> the line numbers (using gdb, e.g. "info line
> >> *stop_machine_create+0x27"), having built the kernel with debug info.
> >>
> >> [top of trace lost due to screen height]
> >> ? sync_page (filemap.c:183)
> >> ? wait_on_page_bit (filemap.c:506)
> >> ? wake_bit_function (wait.c:174)
> >> ? shrink_page_list (vmscan.c:696)
> >> ? __delayacct_blkio_end (delayacct.c:94)
> >> ? finish_wait (list.h:142)
> >> ? congestion_wait (backing-dev.c:761)
> >> ? shrink_inactive_list (vmscan.c:1193)
> >> ? scsi_request_fn (spinlock.h:306)
> >> ? blk_run_queue (blk-core.c:434)
> >> ? shrink_zone (vmscan.c:1484)
> >> ? do_try_to_free_pages (vmscan.c:1684)
> >> ? try_to_free_pages (vmscan.c:1848)
> >> ? isolate_pages_global (vmscan.c:980)
> >> ? __alloc_pages_nodemask (page_alloc.c:1702)
> >> ? __get_free_pages (page_alloc.c:1990)
> >> ? copy_process (fork.c:237)
> >> ? do_fork (fork.c:1443)
> >> ? rb_erase
> >> ? __switch_to
> >> ? kthread
> >> ? kernel_thread
> >> ? kthread
> >> ? kernel_thread_helper
> >> ? kthreadd
> >> ? kthreadd
> >> ? kernel_thread_helper
> >>
> >> INFO: task s2disk:2174 blocked for more than 120 seconds
> >
> > This looks like we have run out of memory while creating a new kernel thread
> > and we have blocked on I/O while trying to free some space (quite obviously,
> > because the I/O doesn't work at this point).
>
> For context, the kernel thread being created here is the stop_machine
> thread. It is created by disable_nonboot_cpus(), called from
> hibernation_snapshot(). See e.g. this hung task backtrace -
>
> http://picasaweb.google.com/lh/photo/BkKUwZCrQ2ceBIM9ZOh7Ow?feat=directlink
>
> > I think it should help if you increase PAGES_FOR_IO, then.
>
> Ok, it's been happening again on 2.6.33-rc6. Unfortunately increasing
> PAGES_FOR_IO doesn't help.
>
> I've been using a test patch to make PAGES_FOR_IO tunable at run time.
> I get the same hang if I increase it by a factor of 10, to 10240:
>
> # cd /sys/module/kernel/parameters/
> # ls
> consoleblank initcall_debug PAGES_FOR_IO panic pause_on_oops SPARE_PAGES
> # echo 10240 > PAGES_FOR_IO
> # echo 2560 > SPARE_PAGES
> # cat SPARE_PAGES
> 2560
> # cat PAGES_FOR_IO
> 10240
>
> I also added a debug patch to try and understand the calculations with
> PAGES_FOR_IO in hibernate_preallocate_memory(). I still don't really
> understand them and there could easily be errors in my debug patch,
> but the output is interesting.
>
> Increasing PAGES_FOR_IO by almost 10000 has the expected effect of
> decreasing "max_size" by the same amount. However it doesn't appear
> to increase the number of free pages at the critical moment.
>
> PAGES_FOR_IO = 1024:
> http://picasaweb.google.com/lh/photo/DYQGvB_4hvCvVuxZf2ibxg?feat=directlink
>
> PAGES_FOR_IO = 10240:
> http://picasaweb.google.com/lh/photo/AIkV_ZBwt22nzN-JdOJCWA?feat=directlink
>
>
> You may remember that I was originally able to avoid the hang by
> reverting commit 5f8dcc2. It doesn't revert cleanly any more.
> However, I tried applying my test&debug patches on top of 5f8dcc2~1
> (just before the commit that triggered the hang). That kernel
> apparently left ~5000 pages free at hibernation time, v.s. ~1200 when
> testing the same scenario on 2.6.33-rc6. (As before, the number of
> free pages remained the same if I increased PAGES_FOR_IO to 10240).
I think the hang may be avoided by using this patch
http://patchwork.kernel.org/patch/74740/
but the hibernation will fail instead.
Can you please repeat your experiments with the patch below applied and
report back?
Rafael
---
kernel/power/snapshot.c | 41 +++++++++++++++++++++++++++++++++++++++--
1 file changed, 39 insertions(+), 2 deletions(-)
Index: linux-2.6/kernel/power/snapshot.c
===================================================================
--- linux-2.6.orig/kernel/power/snapshot.c
+++ linux-2.6/kernel/power/snapshot.c
@@ -1179,6 +1179,10 @@ static void free_unnecessary_pages(void)
to_free_normal -= save_highmem - alloc_highmem;
}
+ printk(KERN_CONT
+ "Freeing %lu normal and %lu highmem preallocated pages\n",
+ to_free_normal, to_free_highmem);
+
memory_bm_position_reset(©_bm);
while (to_free_normal > 0 && to_free_highmem > 0) {
@@ -1300,8 +1304,16 @@ int hibernate_preallocate_memory(void)
/* Compute the maximum number of saveable pages to leave in memory. */
max_size = (count - (size + PAGES_FOR_IO)) / 2 - 2 * SPARE_PAGES;
size = DIV_ROUND_UP(image_size, PAGE_SIZE);
+
+ printk(KERN_CONT "Requested image size: %lu pages\n", size);
+
if (size > max_size)
size = max_size;
+
+ printk(KERN_CONT
+ "count = %lu, highmem = %lu, max_size = %lu, saveable = %lu\n",
+ count, highmem, max_size, saveable);
+
/*
* If the maximum is not less than the current number of saveable pages
* in memory, allocate page frames for the image and we're done.
@@ -1312,10 +1324,14 @@ int hibernate_preallocate_memory(void)
goto out;
}
+ printk(KERN_CONT "Target image size: %lu pages\n", size);
+
/* Estimate the minimum size of the image. */
pages = minimum_image_size(saveable);
- if (size < pages)
- size = min_t(unsigned long, pages, max_size);
+ /* if (size < pages)
+ size = min_t(unsigned long, pages, max_size); */
+
+ printk(KERN_CONT "Minimum image size: %lu pages\n", pages);
/*
* Let the memory management subsystem know that we're going to need a
@@ -1325,6 +1341,9 @@ int hibernate_preallocate_memory(void)
*/
shrink_all_memory(saveable - size);
+ pages = minimum_image_size(count_data_pages() + count_highmem_pages());
+ printk(KERN_CONT "Minimum image size: %lu pages\n", pages);
+
/*
* The number of saveable pages in memory was too high, so apply some
* pressure to decrease it. First, make room for the largest possible
@@ -1334,17 +1353,35 @@ int hibernate_preallocate_memory(void)
*/
pages_highmem = preallocate_image_highmem(highmem / 2);
alloc = (count - max_size) - pages_highmem;
+
+ printk(KERN_CONT "pages_highmem = %lu, alloc = %lu\n",
+ pages_highmem, alloc);
+
pages = preallocate_image_memory(alloc);
if (pages < alloc)
goto err_out;
+
+ printk(KERN_CONT "pages = %lu\n", pages);
+
size = max_size - size;
alloc = size;
+
size = preallocate_highmem_fraction(size, highmem, count);
+
+ printk(KERN_CONT "alloc_highmem = %lu, alloc = %lu\n", size, alloc);
+
pages_highmem += size;
alloc -= size;
pages += preallocate_image_memory(alloc);
+
+ printk(KERN_CONT "pages = %lu\n", pages);
+
pages += pages_highmem;
+ printk(KERN_CONT
+ "pages = %lu, pages_highmem = %lu, count - pages = %lu\n",
+ pages, pages_highmem, count - pages);
+
/*
* We only need as many page frames for the image as there are saveable
* pages in memory, but we have allocated more. Release the excessive
On 2/2/10, Rafael J. Wysocki <[email protected]> wrote:
> On Tuesday 02 February 2010, Alan Jenkins wrote:
>> On 1/2/10, Rafael J. Wysocki <[email protected]> wrote:
>> > On Saturday 02 January 2010, Alan Jenkins wrote:
>> > Hi,
>> >
>> >> I've been suffering from s2disk hangs again. This time, the hangs
>> >> were always before the hibernation image was written out.
>> >>
>> >> They're still frustratingly random. I just started trying to work out
>> >> whether doubling PAGES_FOR_IO makes them go away, but they went away
>> >> on their own again.
>> >>
>> >> I did manage to capture a backtrace with debug info though. Here it
>> >> is for 2.6.33-rc2. (It has also happened on rc1). I was able to get
>> >> the line numbers (using gdb, e.g. "info line
>> >> *stop_machine_create+0x27"), having built the kernel with debug info.
>> >>
>> >> [top of trace lost due to screen height]
>> >> ? sync_page (filemap.c:183)
>> >> ? wait_on_page_bit (filemap.c:506)
>> >> ? wake_bit_function (wait.c:174)
>> >> ? shrink_page_list (vmscan.c:696)
>> >> ? __delayacct_blkio_end (delayacct.c:94)
>> >> ? finish_wait (list.h:142)
>> >> ? congestion_wait (backing-dev.c:761)
>> >> ? shrink_inactive_list (vmscan.c:1193)
>> >> ? scsi_request_fn (spinlock.h:306)
>> >> ? blk_run_queue (blk-core.c:434)
>> >> ? shrink_zone (vmscan.c:1484)
>> >> ? do_try_to_free_pages (vmscan.c:1684)
>> >> ? try_to_free_pages (vmscan.c:1848)
>> >> ? isolate_pages_global (vmscan.c:980)
>> >> ? __alloc_pages_nodemask (page_alloc.c:1702)
>> >> ? __get_free_pages (page_alloc.c:1990)
>> >> ? copy_process (fork.c:237)
>> >> ? do_fork (fork.c:1443)
>> >> ? rb_erase
>> >> ? __switch_to
>> >> ? kthread
>> >> ? kernel_thread
>> >> ? kthread
>> >> ? kernel_thread_helper
>> >> ? kthreadd
>> >> ? kthreadd
>> >> ? kernel_thread_helper
>> >>
>> >> INFO: task s2disk:2174 blocked for more than 120 seconds
>> >
>> > This looks like we have run out of memory while creating a new kernel
>> > thread
>> > and we have blocked on I/O while trying to free some space (quite
>> > obviously,
>> > because the I/O doesn't work at this point).
>>
>> For context, the kernel thread being created here is the stop_machine
>> thread. It is created by disable_nonboot_cpus(), called from
>> hibernation_snapshot(). See e.g. this hung task backtrace -
>>
>> http://picasaweb.google.com/lh/photo/BkKUwZCrQ2ceBIM9ZOh7Ow?feat=directlink
>>
>> > I think it should help if you increase PAGES_FOR_IO, then.
>>
>> Ok, it's been happening again on 2.6.33-rc6. Unfortunately increasing
>> PAGES_FOR_IO doesn't help.
>>
>> I've been using a test patch to make PAGES_FOR_IO tunable at run time.
>> I get the same hang if I increase it by a factor of 10, to 10240:
>>
>> # cd /sys/module/kernel/parameters/
>> # ls
>> consoleblank initcall_debug PAGES_FOR_IO panic pause_on_oops
>> SPARE_PAGES
>> # echo 10240 > PAGES_FOR_IO
>> # echo 2560 > SPARE_PAGES
>> # cat SPARE_PAGES
>> 2560
>> # cat PAGES_FOR_IO
>> 10240
>>
>> I also added a debug patch to try and understand the calculations with
>> PAGES_FOR_IO in hibernate_preallocate_memory(). I still don't really
>> understand them and there could easily be errors in my debug patch,
>> but the output is interesting.
>>
>> Increasing PAGES_FOR_IO by almost 10000 has the expected effect of
>> decreasing "max_size" by the same amount. However it doesn't appear
>> to increase the number of free pages at the critical moment.
>>
>> PAGES_FOR_IO = 1024:
>> http://picasaweb.google.com/lh/photo/DYQGvB_4hvCvVuxZf2ibxg?feat=directlink
>>
>> PAGES_FOR_IO = 10240:
>> http://picasaweb.google.com/lh/photo/AIkV_ZBwt22nzN-JdOJCWA?feat=directlink
>>
>>
>> You may remember that I was originally able to avoid the hang by
>> reverting commit 5f8dcc2. It doesn't revert cleanly any more.
>> However, I tried applying my test&debug patches on top of 5f8dcc2~1
>> (just before the commit that triggered the hang). That kernel
>> apparently left ~5000 pages free at hibernation time, v.s. ~1200 when
>> testing the same scenario on 2.6.33-rc6. (As before, the number of
>> free pages remained the same if I increased PAGES_FOR_IO to 10240).
>
> I think the hang may be avoided by using this patch
> http://patchwork.kernel.org/patch/74740/
> but the hibernation will fail instead.
>
> Can you please repeat your experiments with the patch below applied and
> report back?
>
> Rafael
It causes hibernation to succeed <grin>.
I've attached a dmesg from a successful hibernation with both patches
applied. And for comparison, a screenshot from a hung hibernation
without the fix, but with the debug patch you sent me.
[In both cases I tested directly on top of v2.6.33-rc6, i.e. no
changes to PAGES_FOR_IO or anything else].
Many thanks!
Alan
Alan Jenkins wrote:
> On 2/2/10, Rafael J. Wysocki <[email protected]> wrote:
>
>> On Tuesday 02 February 2010, Alan Jenkins wrote:
>>
>>> On 1/2/10, Rafael J. Wysocki <[email protected]> wrote:
>>>
>>>> On Saturday 02 January 2010, Alan Jenkins wrote:
>>>> Hi,
>>>>
>>>>
>>>>> I've been suffering from s2disk hangs again. This time, the hangs
>>>>> were always before the hibernation image was written out.
>>>>>
>>>>> They're still frustratingly random. I just started trying to work out
>>>>> whether doubling PAGES_FOR_IO makes them go away, but they went away
>>>>> on their own again.
>>>>>
>>>>> I did manage to capture a backtrace with debug info though. Here it
>>>>> is for 2.6.33-rc2. (It has also happened on rc1). I was able to get
>>>>> the line numbers (using gdb, e.g. "info line
>>>>> *stop_machine_create+0x27"), having built the kernel with debug info.
>>>>>
>>>>> [top of trace lost due to screen height]
>>>>> ? sync_page (filemap.c:183)
>>>>> ? wait_on_page_bit (filemap.c:506)
>>>>> ? wake_bit_function (wait.c:174)
>>>>> ? shrink_page_list (vmscan.c:696)
>>>>> ? __delayacct_blkio_end (delayacct.c:94)
>>>>> ? finish_wait (list.h:142)
>>>>> ? congestion_wait (backing-dev.c:761)
>>>>> ? shrink_inactive_list (vmscan.c:1193)
>>>>> ? scsi_request_fn (spinlock.h:306)
>>>>> ? blk_run_queue (blk-core.c:434)
>>>>> ? shrink_zone (vmscan.c:1484)
>>>>> ? do_try_to_free_pages (vmscan.c:1684)
>>>>> ? try_to_free_pages (vmscan.c:1848)
>>>>> ? isolate_pages_global (vmscan.c:980)
>>>>> ? __alloc_pages_nodemask (page_alloc.c:1702)
>>>>> ? __get_free_pages (page_alloc.c:1990)
>>>>> ? copy_process (fork.c:237)
>>>>> ? do_fork (fork.c:1443)
>>>>> ? rb_erase
>>>>> ? __switch_to
>>>>> ? kthread
>>>>> ? kernel_thread
>>>>> ? kthread
>>>>> ? kernel_thread_helper
>>>>> ? kthreadd
>>>>> ? kthreadd
>>>>> ? kernel_thread_helper
>>>>>
>>>>> INFO: task s2disk:2174 blocked for more than 120 seconds
>>>>>
>>>> This looks like we have run out of memory while creating a new kernel
>>>> thread
>>>> and we have blocked on I/O while trying to free some space (quite
>>>> obviously,
>>>> because the I/O doesn't work at this point).
>>>>
>>> For context, the kernel thread being created here is the stop_machine
>>> thread. It is created by disable_nonboot_cpus(), called from
>>> hibernation_snapshot(). See e.g. this hung task backtrace -
>>>
>>> http://picasaweb.google.com/lh/photo/BkKUwZCrQ2ceBIM9ZOh7Ow?feat=directlink
>>>
>>>
>>>> I think it should help if you increase PAGES_FOR_IO, then.
>>>>
>>> Ok, it's been happening again on 2.6.33-rc6. Unfortunately increasing
>>> PAGES_FOR_IO doesn't help.
>>>
>>> I've been using a test patch to make PAGES_FOR_IO tunable at run time.
>>> I get the same hang if I increase it by a factor of 10, to 10240:
>>>
>>> # cd /sys/module/kernel/parameters/
>>> # ls
>>> consoleblank initcall_debug PAGES_FOR_IO panic pause_on_oops
>>> SPARE_PAGES
>>> # echo 10240 > PAGES_FOR_IO
>>> # echo 2560 > SPARE_PAGES
>>> # cat SPARE_PAGES
>>> 2560
>>> # cat PAGES_FOR_IO
>>> 10240
>>>
>>> I also added a debug patch to try and understand the calculations with
>>> PAGES_FOR_IO in hibernate_preallocate_memory(). I still don't really
>>> understand them and there could easily be errors in my debug patch,
>>> but the output is interesting.
>>>
>>> Increasing PAGES_FOR_IO by almost 10000 has the expected effect of
>>> decreasing "max_size" by the same amount. However it doesn't appear
>>> to increase the number of free pages at the critical moment.
>>>
>>> PAGES_FOR_IO = 1024:
>>> http://picasaweb.google.com/lh/photo/DYQGvB_4hvCvVuxZf2ibxg?feat=directlink
>>>
>>> PAGES_FOR_IO = 10240:
>>> http://picasaweb.google.com/lh/photo/AIkV_ZBwt22nzN-JdOJCWA?feat=directlink
>>>
>>>
>>> You may remember that I was originally able to avoid the hang by
>>> reverting commit 5f8dcc2. It doesn't revert cleanly any more.
>>> However, I tried applying my test&debug patches on top of 5f8dcc2~1
>>> (just before the commit that triggered the hang). That kernel
>>> apparently left ~5000 pages free at hibernation time, v.s. ~1200 when
>>> testing the same scenario on 2.6.33-rc6. (As before, the number of
>>> free pages remained the same if I increased PAGES_FOR_IO to 10240).
>>>
>> I think the hang may be avoided by using this patch
>> http://patchwork.kernel.org/patch/74740/
>> but the hibernation will fail instead.
>>
>> Can you please repeat your experiments with the patch below applied and
>> report back?
>>
>> Rafael
>>
>
> It causes hibernation to succeed <grin>.
>
Perhaps I spoke too soon. I see the same hang if I run too many
applications. The first hibernation fails with "not enough swap" as
expected, but the second or third attempt hangs (with the same backtrace
as before).
The patch definitely helps though. Without the patch, I see a hang the
first time I try to hibernate with too many applications running.
Regards
Alan
On Tuesday 09 February 2010, Alan Jenkins wrote:
> Alan Jenkins wrote:
> > On 2/2/10, Rafael J. Wysocki <[email protected]> wrote:
> >
> >> On Tuesday 02 February 2010, Alan Jenkins wrote:
> >>
> >>> On 1/2/10, Rafael J. Wysocki <[email protected]> wrote:
> >>>
> >>>> On Saturday 02 January 2010, Alan Jenkins wrote:
> >>>> Hi,
> >>>>
> >>>>
> >>>>> I've been suffering from s2disk hangs again. This time, the hangs
> >>>>> were always before the hibernation image was written out.
> >>>>>
> >>>>> They're still frustratingly random. I just started trying to work out
> >>>>> whether doubling PAGES_FOR_IO makes them go away, but they went away
> >>>>> on their own again.
> >>>>>
> >>>>> I did manage to capture a backtrace with debug info though. Here it
> >>>>> is for 2.6.33-rc2. (It has also happened on rc1). I was able to get
> >>>>> the line numbers (using gdb, e.g. "info line
> >>>>> *stop_machine_create+0x27"), having built the kernel with debug info.
> >>>>>
> >>>>> [top of trace lost due to screen height]
> >>>>> ? sync_page (filemap.c:183)
> >>>>> ? wait_on_page_bit (filemap.c:506)
> >>>>> ? wake_bit_function (wait.c:174)
> >>>>> ? shrink_page_list (vmscan.c:696)
> >>>>> ? __delayacct_blkio_end (delayacct.c:94)
> >>>>> ? finish_wait (list.h:142)
> >>>>> ? congestion_wait (backing-dev.c:761)
> >>>>> ? shrink_inactive_list (vmscan.c:1193)
> >>>>> ? scsi_request_fn (spinlock.h:306)
> >>>>> ? blk_run_queue (blk-core.c:434)
> >>>>> ? shrink_zone (vmscan.c:1484)
> >>>>> ? do_try_to_free_pages (vmscan.c:1684)
> >>>>> ? try_to_free_pages (vmscan.c:1848)
> >>>>> ? isolate_pages_global (vmscan.c:980)
> >>>>> ? __alloc_pages_nodemask (page_alloc.c:1702)
> >>>>> ? __get_free_pages (page_alloc.c:1990)
> >>>>> ? copy_process (fork.c:237)
> >>>>> ? do_fork (fork.c:1443)
> >>>>> ? rb_erase
> >>>>> ? __switch_to
> >>>>> ? kthread
> >>>>> ? kernel_thread
> >>>>> ? kthread
> >>>>> ? kernel_thread_helper
> >>>>> ? kthreadd
> >>>>> ? kthreadd
> >>>>> ? kernel_thread_helper
> >>>>>
> >>>>> INFO: task s2disk:2174 blocked for more than 120 seconds
> >>>>>
> >>>> This looks like we have run out of memory while creating a new kernel
> >>>> thread
> >>>> and we have blocked on I/O while trying to free some space (quite
> >>>> obviously,
> >>>> because the I/O doesn't work at this point).
> >>>>
> >>> For context, the kernel thread being created here is the stop_machine
> >>> thread. It is created by disable_nonboot_cpus(), called from
> >>> hibernation_snapshot(). See e.g. this hung task backtrace -
> >>>
> >>> http://picasaweb.google.com/lh/photo/BkKUwZCrQ2ceBIM9ZOh7Ow?feat=directlink
> >>>
> >>>
> >>>> I think it should help if you increase PAGES_FOR_IO, then.
> >>>>
> >>> Ok, it's been happening again on 2.6.33-rc6. Unfortunately increasing
> >>> PAGES_FOR_IO doesn't help.
> >>>
> >>> I've been using a test patch to make PAGES_FOR_IO tunable at run time.
> >>> I get the same hang if I increase it by a factor of 10, to 10240:
> >>>
> >>> # cd /sys/module/kernel/parameters/
> >>> # ls
> >>> consoleblank initcall_debug PAGES_FOR_IO panic pause_on_oops
> >>> SPARE_PAGES
> >>> # echo 10240 > PAGES_FOR_IO
> >>> # echo 2560 > SPARE_PAGES
> >>> # cat SPARE_PAGES
> >>> 2560
> >>> # cat PAGES_FOR_IO
> >>> 10240
> >>>
> >>> I also added a debug patch to try and understand the calculations with
> >>> PAGES_FOR_IO in hibernate_preallocate_memory(). I still don't really
> >>> understand them and there could easily be errors in my debug patch,
> >>> but the output is interesting.
> >>>
> >>> Increasing PAGES_FOR_IO by almost 10000 has the expected effect of
> >>> decreasing "max_size" by the same amount. However it doesn't appear
> >>> to increase the number of free pages at the critical moment.
> >>>
> >>> PAGES_FOR_IO = 1024:
> >>> http://picasaweb.google.com/lh/photo/DYQGvB_4hvCvVuxZf2ibxg?feat=directlink
> >>>
> >>> PAGES_FOR_IO = 10240:
> >>> http://picasaweb.google.com/lh/photo/AIkV_ZBwt22nzN-JdOJCWA?feat=directlink
> >>>
> >>>
> >>> You may remember that I was originally able to avoid the hang by
> >>> reverting commit 5f8dcc2. It doesn't revert cleanly any more.
> >>> However, I tried applying my test&debug patches on top of 5f8dcc2~1
> >>> (just before the commit that triggered the hang). That kernel
> >>> apparently left ~5000 pages free at hibernation time, v.s. ~1200 when
> >>> testing the same scenario on 2.6.33-rc6. (As before, the number of
> >>> free pages remained the same if I increased PAGES_FOR_IO to 10240).
> >>>
> >> I think the hang may be avoided by using this patch
> >> http://patchwork.kernel.org/patch/74740/
> >> but the hibernation will fail instead.
> >>
> >> Can you please repeat your experiments with the patch below applied and
> >> report back?
> >>
> >> Rafael
> >>
> >
> > It causes hibernation to succeed <grin>.
> >
>
> Perhaps I spoke too soon. I see the same hang if I run too many
> applications. The first hibernation fails with "not enough swap" as
> expected, but the second or third attempt hangs (with the same backtrace
> as before).
>
> The patch definitely helps though. Without the patch, I see a hang the
> first time I try to hibernate with too many applications running.
Well, I have an idea.
Can you try to apply the appended patch in addition and see if that helps?
Rafael
---
kernel/power/snapshot.c | 11 +++++++++++
1 file changed, 11 insertions(+)
Index: linux-2.6/kernel/power/snapshot.c
===================================================================
--- linux-2.6.orig/kernel/power/snapshot.c
+++ linux-2.6/kernel/power/snapshot.c
@@ -1179,6 +1179,17 @@ static void free_unnecessary_pages(void)
to_free_normal -= save_highmem - alloc_highmem;
}
+ /*
+ * After we have preallocated memory for the image there may be too
+ * little memory for other things done later down the road, like
+ * starting new kernel threads for disabling nonboot CPUs. Try to
+ * mitigate this by reducing the number of pages that we're going to
+ * keep preallocated by 20%.
+ */
+ to_free_normal += (alloc_normal - to_free_normal) / 5;
+ if (to_free_normal > alloc_normal)
+ to_free_normal = alloc_normal;
+
memory_bm_position_reset(©_bm);
while (to_free_normal > 0 && to_free_highmem > 0) {
On 2/15/10, Rafael J. Wysocki <[email protected]> wrote:
> On Tuesday 09 February 2010, Alan Jenkins wrote:
>> Perhaps I spoke too soon. I see the same hang if I run too many
>> applications. The first hibernation fails with "not enough swap" as
>> expected, but the second or third attempt hangs (with the same backtrace
>> as before).
>>
>> The patch definitely helps though. Without the patch, I see a hang the
>> first time I try to hibernate with too many applications running.
>
> Well, I have an idea.
>
> Can you try to apply the appended patch in addition and see if that helps?
>
> Rafael
It doesn't seem to help.
Thanks
Alan
On 2/16/10, Alan Jenkins <[email protected]> wrote:
> On 2/15/10, Rafael J. Wysocki <[email protected]> wrote:
>> On Tuesday 09 February 2010, Alan Jenkins wrote:
>>> Perhaps I spoke too soon. I see the same hang if I run too many
>>> applications. The first hibernation fails with "not enough swap" as
>>> expected, but the second or third attempt hangs (with the same backtrace
>>> as before).
>>>
>>> The patch definitely helps though. Without the patch, I see a hang the
>>> first time I try to hibernate with too many applications running.
>>
>> Well, I have an idea.
>>
>> Can you try to apply the appended patch in addition and see if that
>> helps?
>>
>> Rafael
>
> It doesn't seem to help.
To be clear: It doesn't stop the hang when I hibernate with too many
applications.
It does stop the same hang in a different case though.
1. boot with init=/bin/bash
2. run s2disk
3. cancel the s2disk
4. repeat steps 2&3
With the patch, I can run 10s of iterations, with no hang.
Without the patch, it soon hangs, (in disable_nonboot_cpus(), as always).
That's what happens on 2.6.33-rc7. On 2.6.30, there is no problem.
On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an allocation
failure after a couple of iterations ("kthreadd: page allocation
failure. order:1, mode:0xd0"). It looks like it might be the same
stop_machine thread allocation failure that causes the hang.
Regards
Alan
On Tuesday 16 February 2010, Alan Jenkins wrote:
> On 2/16/10, Alan Jenkins <[email protected]> wrote:
> > On 2/15/10, Rafael J. Wysocki <[email protected]> wrote:
> >> On Tuesday 09 February 2010, Alan Jenkins wrote:
> >>> Perhaps I spoke too soon. I see the same hang if I run too many
> >>> applications. The first hibernation fails with "not enough swap" as
> >>> expected, but the second or third attempt hangs (with the same backtrace
> >>> as before).
> >>>
> >>> The patch definitely helps though. Without the patch, I see a hang the
> >>> first time I try to hibernate with too many applications running.
> >>
> >> Well, I have an idea.
> >>
> >> Can you try to apply the appended patch in addition and see if that
> >> helps?
> >>
> >> Rafael
> >
> > It doesn't seem to help.
>
> To be clear: It doesn't stop the hang when I hibernate with too many
> applications.
>
> It does stop the same hang in a different case though.
>
> 1. boot with init=/bin/bash
> 2. run s2disk
> 3. cancel the s2disk
> 4. repeat steps 2&3
>
> With the patch, I can run 10s of iterations, with no hang.
> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as always).
>
> That's what happens on 2.6.33-rc7. On 2.6.30, there is no problem.
> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an allocation
> failure after a couple of iterations ("kthreadd: page allocation
> failure. order:1, mode:0xd0"). It looks like it might be the same
> stop_machine thread allocation failure that causes the hang.
Have you tested it alone or on top of the previous one? If you've tested it
alone, please apply the appended one in addition to it and retest.
Rafael
---
From: Rafael J. Wysocki <[email protected]>
Subject: MM / PM: Force GFP_NOIO during suspend/hibernation and resume (rev. 3)
There are quite a few GFP_KERNEL memory allocations made during
suspend/hibernation and resume that may cause the system to hang,
because the I/O operations they depend on cannot be completed due to
the underlying devices being suspended.
Avoid this problem by clearing the __GFP_IO and __GFP_FS bits in
gfp_allowed_mask before suspend/hibernation and restoring the
original values of these bits in gfp_allowed_mask durig the
subsequent resume.
Signed-off-by: Rafael J. Wysocki <[email protected]>
Reported-by: Maxim Levitsky <[email protected]>
---
include/linux/gfp.h | 7 +++----
init/main.c | 2 +-
kernel/power/hibernate.c | 9 +++++++++
kernel/power/suspend.c | 3 +++
mm/page_alloc.c | 26 ++++++++++++++++++++++++++
5 files changed, 42 insertions(+), 5 deletions(-)
Index: linux-2.6/include/linux/gfp.h
===================================================================
--- linux-2.6.orig/include/linux/gfp.h
+++ linux-2.6/include/linux/gfp.h
@@ -83,6 +83,7 @@ struct vm_area_struct;
#define GFP_HIGHUSER_MOVABLE (__GFP_WAIT | __GFP_IO | __GFP_FS | \
__GFP_HARDWALL | __GFP_HIGHMEM | \
__GFP_MOVABLE)
+#define GFP_IOFS (__GFP_IO | __GFP_FS)
#ifdef CONFIG_NUMA
#define GFP_THISNODE (__GFP_THISNODE | __GFP_NOWARN | __GFP_NORETRY)
@@ -337,9 +338,7 @@ void drain_local_pages(void *dummy);
extern gfp_t gfp_allowed_mask;
-static inline void set_gfp_allowed_mask(gfp_t mask)
-{
- gfp_allowed_mask = mask;
-}
+extern void set_gfp_allowed_mask(gfp_t mask);
+extern gfp_t clear_gfp_allowed_mask(gfp_t mask);
#endif /* __LINUX_GFP_H */
Index: linux-2.6/init/main.c
===================================================================
--- linux-2.6.orig/init/main.c
+++ linux-2.6/init/main.c
@@ -601,7 +601,7 @@ asmlinkage void __init start_kernel(void
local_irq_enable();
/* Interrupts are enabled now so all GFP allocations are safe. */
- set_gfp_allowed_mask(__GFP_BITS_MASK);
+ gfp_allowed_mask = __GFP_BITS_MASK;
kmem_cache_init_late();
Index: linux-2.6/mm/page_alloc.c
===================================================================
--- linux-2.6.orig/mm/page_alloc.c
+++ linux-2.6/mm/page_alloc.c
@@ -76,6 +76,32 @@ unsigned long totalreserve_pages __read_
int percpu_pagelist_fraction;
gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK;
+#ifdef CONFIG_PM_SLEEP
+/*
+ * The following functions are used by the suspend/hibernate code to temporarily
+ * change gfp_allowed_mask in order to avoid using I/O during memory allocations
+ * while devices are suspended. To avoid races with the suspend/hibernate code,
+ * they should always be called with pm_mutex held (gfp_allowed_mask also should
+ * only be modified with pm_mutex held, unless the suspend/hibernate code is
+ * guaranteed not to run in parallel with that modification).
+ */
+
+void set_gfp_allowed_mask(gfp_t mask)
+{
+ WARN_ON(!mutex_is_locked(&pm_mutex));
+ gfp_allowed_mask = mask;
+}
+
+gfp_t clear_gfp_allowed_mask(gfp_t mask)
+{
+ gfp_t ret = gfp_allowed_mask;
+
+ WARN_ON(!mutex_is_locked(&pm_mutex));
+ gfp_allowed_mask &= ~mask;
+ return ret;
+}
+#endif /* CONFIG_PM_SLEEP */
+
#ifdef CONFIG_HUGETLB_PAGE_SIZE_VARIABLE
int pageblock_order __read_mostly;
#endif
Index: linux-2.6/kernel/power/hibernate.c
===================================================================
--- linux-2.6.orig/kernel/power/hibernate.c
+++ linux-2.6/kernel/power/hibernate.c
@@ -323,6 +323,7 @@ static int create_image(int platform_mod
int hibernation_snapshot(int platform_mode)
{
int error;
+ gfp_t saved_mask;
error = platform_begin(platform_mode);
if (error)
@@ -334,6 +335,7 @@ int hibernation_snapshot(int platform_mo
goto Close;
suspend_console();
+ saved_mask = clear_gfp_allowed_mask(GFP_IOFS);
error = dpm_suspend_start(PMSG_FREEZE);
if (error)
goto Recover_platform;
@@ -351,6 +353,7 @@ int hibernation_snapshot(int platform_mo
dpm_resume_end(in_suspend ?
(error ? PMSG_RECOVER : PMSG_THAW) : PMSG_RESTORE);
+ set_gfp_allowed_mask(saved_mask);
resume_console();
Close:
platform_end(platform_mode);
@@ -445,14 +448,17 @@ static int resume_target_kernel(bool pla
int hibernation_restore(int platform_mode)
{
int error;
+ gfp_t saved_mask;
pm_prepare_console();
suspend_console();
+ saved_mask = clear_gfp_allowed_mask(GFP_IOFS);
error = dpm_suspend_start(PMSG_QUIESCE);
if (!error) {
error = resume_target_kernel(platform_mode);
dpm_resume_end(PMSG_RECOVER);
}
+ set_gfp_allowed_mask(saved_mask);
resume_console();
pm_restore_console();
return error;
@@ -466,6 +472,7 @@ int hibernation_restore(int platform_mod
int hibernation_platform_enter(void)
{
int error;
+ gfp_t saved_mask;
if (!hibernation_ops)
return -ENOSYS;
@@ -481,6 +488,7 @@ int hibernation_platform_enter(void)
entering_platform_hibernation = true;
suspend_console();
+ saved_mask = clear_gfp_allowed_mask(GFP_IOFS);
error = dpm_suspend_start(PMSG_HIBERNATE);
if (error) {
if (hibernation_ops->recover)
@@ -518,6 +526,7 @@ int hibernation_platform_enter(void)
Resume_devices:
entering_platform_hibernation = false;
dpm_resume_end(PMSG_RESTORE);
+ set_gfp_allowed_mask(saved_mask);
resume_console();
Close:
Index: linux-2.6/kernel/power/suspend.c
===================================================================
--- linux-2.6.orig/kernel/power/suspend.c
+++ linux-2.6/kernel/power/suspend.c
@@ -198,6 +198,7 @@ static int suspend_enter(suspend_state_t
int suspend_devices_and_enter(suspend_state_t state)
{
int error;
+ gfp_t saved_mask;
if (!suspend_ops)
return -ENOSYS;
@@ -208,6 +209,7 @@ int suspend_devices_and_enter(suspend_st
goto Close;
}
suspend_console();
+ saved_mask = clear_gfp_allowed_mask(GFP_IOFS);
suspend_test_start();
error = dpm_suspend_start(PMSG_SUSPEND);
if (error) {
@@ -224,6 +226,7 @@ int suspend_devices_and_enter(suspend_st
suspend_test_start();
dpm_resume_end(PMSG_RESUME);
suspend_test_finish("resume devices");
+ set_gfp_allowed_mask(saved_mask);
resume_console();
Close:
if (suspend_ops->end)
On 2/16/10, Rafael J. Wysocki <[email protected]> wrote:
> On Tuesday 16 February 2010, Alan Jenkins wrote:
>> On 2/16/10, Alan Jenkins <[email protected]> wrote:
>> > On 2/15/10, Rafael J. Wysocki <[email protected]> wrote:
>> >> On Tuesday 09 February 2010, Alan Jenkins wrote:
>> >>> Perhaps I spoke too soon. I see the same hang if I run too many
>> >>> applications. The first hibernation fails with "not enough swap" as
>> >>> expected, but the second or third attempt hangs (with the same
>> >>> backtrace
>> >>> as before).
>> >>>
>> >>> The patch definitely helps though. Without the patch, I see a hang
>> >>> the
>> >>> first time I try to hibernate with too many applications running.
>> >>
>> >> Well, I have an idea.
>> >>
>> >> Can you try to apply the appended patch in addition and see if that
>> >> helps?
>> >>
>> >> Rafael
>> >
>> > It doesn't seem to help.
>>
>> To be clear: It doesn't stop the hang when I hibernate with too many
>> applications.
>>
>> It does stop the same hang in a different case though.
>>
>> 1. boot with init=/bin/bash
>> 2. run s2disk
>> 3. cancel the s2disk
>> 4. repeat steps 2&3
>>
>> With the patch, I can run 10s of iterations, with no hang.
>> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as always).
>>
>> That's what happens on 2.6.33-rc7. On 2.6.30, there is no problem.
>> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an allocation
>> failure after a couple of iterations ("kthreadd: page allocation
>> failure. order:1, mode:0xd0"). It looks like it might be the same
>> stop_machine thread allocation failure that causes the hang.
>
> Have you tested it alone or on top of the previous one? If you've tested it
> alone, please apply the appended one in addition to it and retest.
>
> Rafael
I did test with both patches applied together -
1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation and resume
2. "reducing the number of pages that we're going to keep preallocated by 20%"
Alan
On Wednesday 17 February 2010, Alan Jenkins wrote:
> On 2/16/10, Rafael J. Wysocki <[email protected]> wrote:
> > On Tuesday 16 February 2010, Alan Jenkins wrote:
> >> On 2/16/10, Alan Jenkins <[email protected]> wrote:
> >> > On 2/15/10, Rafael J. Wysocki <[email protected]> wrote:
> >> >> On Tuesday 09 February 2010, Alan Jenkins wrote:
> >> >>> Perhaps I spoke too soon. I see the same hang if I run too many
> >> >>> applications. The first hibernation fails with "not enough swap" as
> >> >>> expected, but the second or third attempt hangs (with the same
> >> >>> backtrace
> >> >>> as before).
> >> >>>
> >> >>> The patch definitely helps though. Without the patch, I see a hang
> >> >>> the
> >> >>> first time I try to hibernate with too many applications running.
> >> >>
> >> >> Well, I have an idea.
> >> >>
> >> >> Can you try to apply the appended patch in addition and see if that
> >> >> helps?
> >> >>
> >> >> Rafael
> >> >
> >> > It doesn't seem to help.
> >>
> >> To be clear: It doesn't stop the hang when I hibernate with too many
> >> applications.
> >>
> >> It does stop the same hang in a different case though.
> >>
> >> 1. boot with init=/bin/bash
> >> 2. run s2disk
> >> 3. cancel the s2disk
> >> 4. repeat steps 2&3
> >>
> >> With the patch, I can run 10s of iterations, with no hang.
> >> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as always).
> >>
> >> That's what happens on 2.6.33-rc7. On 2.6.30, there is no problem.
> >> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an allocation
> >> failure after a couple of iterations ("kthreadd: page allocation
> >> failure. order:1, mode:0xd0"). It looks like it might be the same
> >> stop_machine thread allocation failure that causes the hang.
> >
> > Have you tested it alone or on top of the previous one? If you've tested it
> > alone, please apply the appended one in addition to it and retest.
> >
> > Rafael
>
> I did test with both patches applied together -
>
> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation and resume
> 2. "reducing the number of pages that we're going to keep preallocated by 20%"
In that case you can try to reduce the number of preallocated pages even more,
ie. change "/ 5" to "/ 2" (for example) in the second patch.
Rafael
On 2/17/10, Rafael J. Wysocki <[email protected]> wrote:
> On Wednesday 17 February 2010, Alan Jenkins wrote:
>> On 2/16/10, Rafael J. Wysocki <[email protected]> wrote:
>> > On Tuesday 16 February 2010, Alan Jenkins wrote:
>> >> On 2/16/10, Alan Jenkins <[email protected]> wrote:
>> >> > On 2/15/10, Rafael J. Wysocki <[email protected]> wrote:
>> >> >> On Tuesday 09 February 2010, Alan Jenkins wrote:
>> >> >>> Perhaps I spoke too soon. I see the same hang if I run too many
>> >> >>> applications. The first hibernation fails with "not enough swap"
>> >> >>> as
>> >> >>> expected, but the second or third attempt hangs (with the same
>> >> >>> backtrace
>> >> >>> as before).
>> >> >>>
>> >> >>> The patch definitely helps though. Without the patch, I see a hang
>> >> >>> the
>> >> >>> first time I try to hibernate with too many applications running.
>> >> >>
>> >> >> Well, I have an idea.
>> >> >>
>> >> >> Can you try to apply the appended patch in addition and see if that
>> >> >> helps?
>> >> >>
>> >> >> Rafael
>> >> >
>> >> > It doesn't seem to help.
>> >>
>> >> To be clear: It doesn't stop the hang when I hibernate with too many
>> >> applications.
>> >>
>> >> It does stop the same hang in a different case though.
>> >>
>> >> 1. boot with init=/bin/bash
>> >> 2. run s2disk
>> >> 3. cancel the s2disk
>> >> 4. repeat steps 2&3
>> >>
>> >> With the patch, I can run 10s of iterations, with no hang.
>> >> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as
>> >> always).
>> >>
>> >> That's what happens on 2.6.33-rc7. On 2.6.30, there is no problem.
>> >> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an allocation
>> >> failure after a couple of iterations ("kthreadd: page allocation
>> >> failure. order:1, mode:0xd0"). It looks like it might be the same
>> >> stop_machine thread allocation failure that causes the hang.
>> >
>> > Have you tested it alone or on top of the previous one? If you've
>> > tested it
>> > alone, please apply the appended one in addition to it and retest.
>> >
>> > Rafael
>>
>> I did test with both patches applied together -
>>
>> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation and resume
>> 2. "reducing the number of pages that we're going to keep preallocated by
>> 20%"
>
> In that case you can try to reduce the number of preallocated pages even
> more,
> ie. change "/ 5" to "/ 2" (for example) in the second patch.
It still hangs if I try to hibernate a couple of times with too many
applications.
Alan
On Thursday 18 February 2010, Alan Jenkins wrote:
> On 2/17/10, Rafael J. Wysocki <[email protected]> wrote:
> > On Wednesday 17 February 2010, Alan Jenkins wrote:
> >> On 2/16/10, Rafael J. Wysocki <[email protected]> wrote:
> >> > On Tuesday 16 February 2010, Alan Jenkins wrote:
> >> >> On 2/16/10, Alan Jenkins <[email protected]> wrote:
> >> >> > On 2/15/10, Rafael J. Wysocki <[email protected]> wrote:
> >> >> >> On Tuesday 09 February 2010, Alan Jenkins wrote:
> >> >> >>> Perhaps I spoke too soon. I see the same hang if I run too many
> >> >> >>> applications. The first hibernation fails with "not enough swap"
> >> >> >>> as
> >> >> >>> expected, but the second or third attempt hangs (with the same
> >> >> >>> backtrace
> >> >> >>> as before).
> >> >> >>>
> >> >> >>> The patch definitely helps though. Without the patch, I see a hang
> >> >> >>> the
> >> >> >>> first time I try to hibernate with too many applications running.
> >> >> >>
> >> >> >> Well, I have an idea.
> >> >> >>
> >> >> >> Can you try to apply the appended patch in addition and see if that
> >> >> >> helps?
> >> >> >>
> >> >> >> Rafael
> >> >> >
> >> >> > It doesn't seem to help.
> >> >>
> >> >> To be clear: It doesn't stop the hang when I hibernate with too many
> >> >> applications.
> >> >>
> >> >> It does stop the same hang in a different case though.
> >> >>
> >> >> 1. boot with init=/bin/bash
> >> >> 2. run s2disk
> >> >> 3. cancel the s2disk
> >> >> 4. repeat steps 2&3
> >> >>
> >> >> With the patch, I can run 10s of iterations, with no hang.
> >> >> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as
> >> >> always).
> >> >>
> >> >> That's what happens on 2.6.33-rc7. On 2.6.30, there is no problem.
> >> >> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an allocation
> >> >> failure after a couple of iterations ("kthreadd: page allocation
> >> >> failure. order:1, mode:0xd0"). It looks like it might be the same
> >> >> stop_machine thread allocation failure that causes the hang.
> >> >
> >> > Have you tested it alone or on top of the previous one? If you've
> >> > tested it
> >> > alone, please apply the appended one in addition to it and retest.
> >> >
> >> > Rafael
> >>
> >> I did test with both patches applied together -
> >>
> >> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation and resume
> >> 2. "reducing the number of pages that we're going to keep preallocated by
> >> 20%"
> >
> > In that case you can try to reduce the number of preallocated pages even
> > more,
> > ie. change "/ 5" to "/ 2" (for example) in the second patch.
>
> It still hangs if I try to hibernate a couple of times with too many
> applications.
Hmm. I guess I asked that before, but is this a 32-bit or 64-bit system and
how much RAM is there in the box?
Rafael
On 2/18/10, Rafael J. Wysocki <[email protected]> wrote:
> On Thursday 18 February 2010, Alan Jenkins wrote:
>> On 2/17/10, Rafael J. Wysocki <[email protected]> wrote:
>> > On Wednesday 17 February 2010, Alan Jenkins wrote:
>> >> On 2/16/10, Rafael J. Wysocki <[email protected]> wrote:
>> >> > On Tuesday 16 February 2010, Alan Jenkins wrote:
>> >> >> On 2/16/10, Alan Jenkins <[email protected]> wrote:
>> >> >> > On 2/15/10, Rafael J. Wysocki <[email protected]> wrote:
>> >> >> >> On Tuesday 09 February 2010, Alan Jenkins wrote:
>> >> >> >>> Perhaps I spoke too soon. I see the same hang if I run too many
>> >> >> >>> applications. The first hibernation fails with "not enough
>> >> >> >>> swap"
>> >> >> >>> as
>> >> >> >>> expected, but the second or third attempt hangs (with the same
>> >> >> >>> backtrace
>> >> >> >>> as before).
>> >> >> >>>
>> >> >> >>> The patch definitely helps though. Without the patch, I see a
>> >> >> >>> hang
>> >> >> >>> the
>> >> >> >>> first time I try to hibernate with too many applications
>> >> >> >>> running.
>> >> >> >>
>> >> >> >> Well, I have an idea.
>> >> >> >>
>> >> >> >> Can you try to apply the appended patch in addition and see if
>> >> >> >> that
>> >> >> >> helps?
>> >> >> >>
>> >> >> >> Rafael
>> >> >> >
>> >> >> > It doesn't seem to help.
>> >> >>
>> >> >> To be clear: It doesn't stop the hang when I hibernate with too many
>> >> >> applications.
>> >> >>
>> >> >> It does stop the same hang in a different case though.
>> >> >>
>> >> >> 1. boot with init=/bin/bash
>> >> >> 2. run s2disk
>> >> >> 3. cancel the s2disk
>> >> >> 4. repeat steps 2&3
>> >> >>
>> >> >> With the patch, I can run 10s of iterations, with no hang.
>> >> >> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as
>> >> >> always).
>> >> >>
>> >> >> That's what happens on 2.6.33-rc7. On 2.6.30, there is no problem.
>> >> >> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an
>> >> >> allocation
>> >> >> failure after a couple of iterations ("kthreadd: page allocation
>> >> >> failure. order:1, mode:0xd0"). It looks like it might be the same
>> >> >> stop_machine thread allocation failure that causes the hang.
>> >> >
>> >> > Have you tested it alone or on top of the previous one? If you've
>> >> > tested it
>> >> > alone, please apply the appended one in addition to it and retest.
>> >> >
>> >> > Rafael
>> >>
>> >> I did test with both patches applied together -
>> >>
>> >> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation and
>> >> resume
>> >> 2. "reducing the number of pages that we're going to keep preallocated
>> >> by
>> >> 20%"
>> >
>> > In that case you can try to reduce the number of preallocated pages even
>> > more,
>> > ie. change "/ 5" to "/ 2" (for example) in the second patch.
>>
>> It still hangs if I try to hibernate a couple of times with too many
>> applications.
>
> Hmm. I guess I asked that before, but is this a 32-bit or 64-bit system and
> how much RAM is there in the box?
>
> Rafael
EeePC 701. 32 bit. 512Mb RAM. 350Mb swap file, on a "first-gen" SSD.
Alan
On Friday 19 February 2010, Alan Jenkins wrote:
> On 2/18/10, Rafael J. Wysocki <[email protected]> wrote:
> > On Thursday 18 February 2010, Alan Jenkins wrote:
> >> On 2/17/10, Rafael J. Wysocki <[email protected]> wrote:
> >> > On Wednesday 17 February 2010, Alan Jenkins wrote:
> >> >> On 2/16/10, Rafael J. Wysocki <[email protected]> wrote:
> >> >> > On Tuesday 16 February 2010, Alan Jenkins wrote:
> >> >> >> On 2/16/10, Alan Jenkins <[email protected]> wrote:
> >> >> >> > On 2/15/10, Rafael J. Wysocki <[email protected]> wrote:
> >> >> >> >> On Tuesday 09 February 2010, Alan Jenkins wrote:
> >> >> >> >>> Perhaps I spoke too soon. I see the same hang if I run too many
> >> >> >> >>> applications. The first hibernation fails with "not enough
> >> >> >> >>> swap"
> >> >> >> >>> as
> >> >> >> >>> expected, but the second or third attempt hangs (with the same
> >> >> >> >>> backtrace
> >> >> >> >>> as before).
> >> >> >> >>>
> >> >> >> >>> The patch definitely helps though. Without the patch, I see a
> >> >> >> >>> hang
> >> >> >> >>> the
> >> >> >> >>> first time I try to hibernate with too many applications
> >> >> >> >>> running.
> >> >> >> >>
> >> >> >> >> Well, I have an idea.
> >> >> >> >>
> >> >> >> >> Can you try to apply the appended patch in addition and see if
> >> >> >> >> that
> >> >> >> >> helps?
> >> >> >> >>
> >> >> >> >> Rafael
> >> >> >> >
> >> >> >> > It doesn't seem to help.
> >> >> >>
> >> >> >> To be clear: It doesn't stop the hang when I hibernate with too many
> >> >> >> applications.
> >> >> >>
> >> >> >> It does stop the same hang in a different case though.
> >> >> >>
> >> >> >> 1. boot with init=/bin/bash
> >> >> >> 2. run s2disk
> >> >> >> 3. cancel the s2disk
> >> >> >> 4. repeat steps 2&3
> >> >> >>
> >> >> >> With the patch, I can run 10s of iterations, with no hang.
> >> >> >> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as
> >> >> >> always).
> >> >> >>
> >> >> >> That's what happens on 2.6.33-rc7. On 2.6.30, there is no problem.
> >> >> >> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an
> >> >> >> allocation
> >> >> >> failure after a couple of iterations ("kthreadd: page allocation
> >> >> >> failure. order:1, mode:0xd0"). It looks like it might be the same
> >> >> >> stop_machine thread allocation failure that causes the hang.
> >> >> >
> >> >> > Have you tested it alone or on top of the previous one? If you've
> >> >> > tested it
> >> >> > alone, please apply the appended one in addition to it and retest.
> >> >> >
> >> >> > Rafael
> >> >>
> >> >> I did test with both patches applied together -
> >> >>
> >> >> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation and
> >> >> resume
> >> >> 2. "reducing the number of pages that we're going to keep preallocated
> >> >> by
> >> >> 20%"
> >> >
> >> > In that case you can try to reduce the number of preallocated pages even
> >> > more,
> >> > ie. change "/ 5" to "/ 2" (for example) in the second patch.
> >>
> >> It still hangs if I try to hibernate a couple of times with too many
> >> applications.
> >
> > Hmm. I guess I asked that before, but is this a 32-bit or 64-bit system and
> > how much RAM is there in the box?
> >
> > Rafael
>
> EeePC 701. 32 bit. 512Mb RAM. 350Mb swap file, on a "first-gen" SSD.
Hmm. I'd try to make free_unnecessary_pages() free all of the preallocated
pages and see what happens.
Rafael
Rafael J. Wysocki wrote:
> On Friday 19 February 2010, Alan Jenkins wrote:
>
>> On 2/18/10, Rafael J. Wysocki <[email protected]> wrote:
>>
>>> On Thursday 18 February 2010, Alan Jenkins wrote:
>>>
>>>> On 2/17/10, Rafael J. Wysocki <[email protected]> wrote:
>>>>
>>>>> On Wednesday 17 February 2010, Alan Jenkins wrote:
>>>>>
>>>>>> On 2/16/10, Rafael J. Wysocki <[email protected]> wrote:
>>>>>>
>>>>>>> On Tuesday 16 February 2010, Alan Jenkins wrote:
>>>>>>>
>>>>>>>> On 2/16/10, Alan Jenkins <[email protected]> wrote:
>>>>>>>>
>>>>>>>>> On 2/15/10, Rafael J. Wysocki <[email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> On Tuesday 09 February 2010, Alan Jenkins wrote:
>>>>>>>>>>
>>>>>>>>>>> Perhaps I spoke too soon. I see the same hang if I run too many
>>>>>>>>>>> applications. The first hibernation fails with "not enough
>>>>>>>>>>> swap"
>>>>>>>>>>> as
>>>>>>>>>>> expected, but the second or third attempt hangs (with the same
>>>>>>>>>>> backtrace
>>>>>>>>>>> as before).
>>>>>>>>>>>
>>>>>>>>>>> The patch definitely helps though. Without the patch, I see a
>>>>>>>>>>> hang
>>>>>>>>>>> the
>>>>>>>>>>> first time I try to hibernate with too many applications
>>>>>>>>>>> running.
>>>>>>>>>>>
>>>>>>>>>> Well, I have an idea.
>>>>>>>>>>
>>>>>>>>>> Can you try to apply the appended patch in addition and see if
>>>>>>>>>> that
>>>>>>>>>> helps?
>>>>>>>>>>
>>>>>>>>>> Rafael
>>>>>>>>>>
>>>>>>>>> It doesn't seem to help.
>>>>>>>>>
>>>>>>>> To be clear: It doesn't stop the hang when I hibernate with too many
>>>>>>>> applications.
>>>>>>>>
>>>>>>>> It does stop the same hang in a different case though.
>>>>>>>>
>>>>>>>> 1. boot with init=/bin/bash
>>>>>>>> 2. run s2disk
>>>>>>>> 3. cancel the s2disk
>>>>>>>> 4. repeat steps 2&3
>>>>>>>>
>>>>>>>> With the patch, I can run 10s of iterations, with no hang.
>>>>>>>> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as
>>>>>>>> always).
>>>>>>>>
>>>>>>>> That's what happens on 2.6.33-rc7. On 2.6.30, there is no problem.
>>>>>>>> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an
>>>>>>>> allocation
>>>>>>>> failure after a couple of iterations ("kthreadd: page allocation
>>>>>>>> failure. order:1, mode:0xd0"). It looks like it might be the same
>>>>>>>> stop_machine thread allocation failure that causes the hang.
>>>>>>>>
>>>>>>> Have you tested it alone or on top of the previous one? If you've
>>>>>>> tested it
>>>>>>> alone, please apply the appended one in addition to it and retest.
>>>>>>>
>>>>>>> Rafael
>>>>>>>
>>>>>> I did test with both patches applied together -
>>>>>>
>>>>>> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation and
>>>>>> resume
>>>>>> 2. "reducing the number of pages that we're going to keep preallocated
>>>>>> by
>>>>>> 20%"
>>>>>>
>>>>> In that case you can try to reduce the number of preallocated pages even
>>>>> more,
>>>>> ie. change "/ 5" to "/ 2" (for example) in the second patch.
>>>>>
>>>> It still hangs if I try to hibernate a couple of times with too many
>>>> applications.
>>>>
>>> Hmm. I guess I asked that before, but is this a 32-bit or 64-bit system and
>>> how much RAM is there in the box?
>>>
>>> Rafael
>>>
>> EeePC 701. 32 bit. 512Mb RAM. 350Mb swap file, on a "first-gen" SSD.
>>
>
> Hmm. I'd try to make free_unnecessary_pages() free all of the preallocated
> pages and see what happens.
>
It still hangs in hibernation_snapshot() / disable_nonboot_cpus().
After apparently freeing over 400Mb / 100,000 pages of preallocated ram.
There is a change which I missed before. When I applied your first
patch ("Force GFP_NOIO during suspend" etc.), it did change the hung
task backtraces a bit. I don't know if it tells us anything.
Without the patch, there were two backtraces. The first backtrace
suggested a problem allocating pages for a kernel thread (at
copy_process() / try_to_free_pages()). The second showed that this
problem was blocking s2disk (at hibernation_snapshot() /
disable_nonboot_cpus() / stop_machine_create()).
With the GFP_NOIO patch, I see only the s2disk backtrace.
Thanks
Alan
On Monday 22 February 2010, Alan Jenkins wrote:
> Rafael J. Wysocki wrote:
> > On Friday 19 February 2010, Alan Jenkins wrote:
> >
> >> On 2/18/10, Rafael J. Wysocki <[email protected]> wrote:
> >>
> >>> On Thursday 18 February 2010, Alan Jenkins wrote:
> >>>
> >>>> On 2/17/10, Rafael J. Wysocki <[email protected]> wrote:
> >>>>
> >>>>> On Wednesday 17 February 2010, Alan Jenkins wrote:
> >>>>>
> >>>>>> On 2/16/10, Rafael J. Wysocki <[email protected]> wrote:
> >>>>>>
> >>>>>>> On Tuesday 16 February 2010, Alan Jenkins wrote:
> >>>>>>>
> >>>>>>>> On 2/16/10, Alan Jenkins <[email protected]> wrote:
> >>>>>>>>
> >>>>>>>>> On 2/15/10, Rafael J. Wysocki <[email protected]> wrote:
> >>>>>>>>>
> >>>>>>>>>> On Tuesday 09 February 2010, Alan Jenkins wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Perhaps I spoke too soon. I see the same hang if I run too many
> >>>>>>>>>>> applications. The first hibernation fails with "not enough
> >>>>>>>>>>> swap"
> >>>>>>>>>>> as
> >>>>>>>>>>> expected, but the second or third attempt hangs (with the same
> >>>>>>>>>>> backtrace
> >>>>>>>>>>> as before).
> >>>>>>>>>>>
> >>>>>>>>>>> The patch definitely helps though. Without the patch, I see a
> >>>>>>>>>>> hang
> >>>>>>>>>>> the
> >>>>>>>>>>> first time I try to hibernate with too many applications
> >>>>>>>>>>> running.
> >>>>>>>>>>>
> >>>>>>>>>> Well, I have an idea.
> >>>>>>>>>>
> >>>>>>>>>> Can you try to apply the appended patch in addition and see if
> >>>>>>>>>> that
> >>>>>>>>>> helps?
> >>>>>>>>>>
> >>>>>>>>>> Rafael
> >>>>>>>>>>
> >>>>>>>>> It doesn't seem to help.
> >>>>>>>>>
> >>>>>>>> To be clear: It doesn't stop the hang when I hibernate with too many
> >>>>>>>> applications.
> >>>>>>>>
> >>>>>>>> It does stop the same hang in a different case though.
> >>>>>>>>
> >>>>>>>> 1. boot with init=/bin/bash
> >>>>>>>> 2. run s2disk
> >>>>>>>> 3. cancel the s2disk
> >>>>>>>> 4. repeat steps 2&3
> >>>>>>>>
> >>>>>>>> With the patch, I can run 10s of iterations, with no hang.
> >>>>>>>> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as
> >>>>>>>> always).
> >>>>>>>>
> >>>>>>>> That's what happens on 2.6.33-rc7. On 2.6.30, there is no problem.
> >>>>>>>> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an
> >>>>>>>> allocation
> >>>>>>>> failure after a couple of iterations ("kthreadd: page allocation
> >>>>>>>> failure. order:1, mode:0xd0"). It looks like it might be the same
> >>>>>>>> stop_machine thread allocation failure that causes the hang.
> >>>>>>>>
> >>>>>>> Have you tested it alone or on top of the previous one? If you've
> >>>>>>> tested it
> >>>>>>> alone, please apply the appended one in addition to it and retest.
> >>>>>>>
> >>>>>>> Rafael
> >>>>>>>
> >>>>>> I did test with both patches applied together -
> >>>>>>
> >>>>>> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation and
> >>>>>> resume
> >>>>>> 2. "reducing the number of pages that we're going to keep preallocated
> >>>>>> by
> >>>>>> 20%"
> >>>>>>
> >>>>> In that case you can try to reduce the number of preallocated pages even
> >>>>> more,
> >>>>> ie. change "/ 5" to "/ 2" (for example) in the second patch.
> >>>>>
> >>>> It still hangs if I try to hibernate a couple of times with too many
> >>>> applications.
> >>>>
> >>> Hmm. I guess I asked that before, but is this a 32-bit or 64-bit system and
> >>> how much RAM is there in the box?
> >>>
> >>> Rafael
> >>>
> >> EeePC 701. 32 bit. 512Mb RAM. 350Mb swap file, on a "first-gen" SSD.
> >>
> >
> > Hmm. I'd try to make free_unnecessary_pages() free all of the preallocated
> > pages and see what happens.
> >
>
> It still hangs in hibernation_snapshot() / disable_nonboot_cpus().
> After apparently freeing over 400Mb / 100,000 pages of preallocated ram.
>
>
>
> There is a change which I missed before. When I applied your first
> patch ("Force GFP_NOIO during suspend" etc.), it did change the hung
> task backtraces a bit. I don't know if it tells us anything.
>
> Without the patch, there were two backtraces. The first backtrace
> suggested a problem allocating pages for a kernel thread (at
> copy_process() / try_to_free_pages()). The second showed that this
> problem was blocking s2disk (at hibernation_snapshot() /
> disable_nonboot_cpus() / stop_machine_create()).
>
> With the GFP_NOIO patch, I see only the s2disk backtrace.
Can you please post this backtrace?
Rafael
On 2/22/10, Rafael J. Wysocki <[email protected]> wrote:
> On Monday 22 February 2010, Alan Jenkins wrote:
>> Rafael J. Wysocki wrote:
>> > On Friday 19 February 2010, Alan Jenkins wrote:
>> >
>> >> On 2/18/10, Rafael J. Wysocki <[email protected]> wrote:
>> >>
>> >>> On Thursday 18 February 2010, Alan Jenkins wrote:
>> >>>
>> >>>> On 2/17/10, Rafael J. Wysocki <[email protected]> wrote:
>> >>>>
>> >>>>> On Wednesday 17 February 2010, Alan Jenkins wrote:
>> >>>>>
>> >>>>>> On 2/16/10, Rafael J. Wysocki <[email protected]> wrote:
>> >>>>>>
>> >>>>>>> On Tuesday 16 February 2010, Alan Jenkins wrote:
>> >>>>>>>
>> >>>>>>>> On 2/16/10, Alan Jenkins <[email protected]> wrote:
>> >>>>>>>>
>> >>>>>>>>> On 2/15/10, Rafael J. Wysocki <[email protected]> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> On Tuesday 09 February 2010, Alan Jenkins wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Perhaps I spoke too soon. I see the same hang if I run too
>> >>>>>>>>>>> many
>> >>>>>>>>>>> applications. The first hibernation fails with "not enough
>> >>>>>>>>>>> swap"
>> >>>>>>>>>>> as
>> >>>>>>>>>>> expected, but the second or third attempt hangs (with the same
>> >>>>>>>>>>> backtrace
>> >>>>>>>>>>> as before).
>> >>>>>>>>>>>
>> >>>>>>>>>>> The patch definitely helps though. Without the patch, I see a
>> >>>>>>>>>>> hang
>> >>>>>>>>>>> the
>> >>>>>>>>>>> first time I try to hibernate with too many applications
>> >>>>>>>>>>> running.
>> >>>>>>>>>>>
>> >>>>>>>>>> Well, I have an idea.
>> >>>>>>>>>>
>> >>>>>>>>>> Can you try to apply the appended patch in addition and see if
>> >>>>>>>>>> that
>> >>>>>>>>>> helps?
>> >>>>>>>>>>
>> >>>>>>>>>> Rafael
>> >>>>>>>>>>
>> >>>>>>>>> It doesn't seem to help.
>> >>>>>>>>>
>> >>>>>>>> To be clear: It doesn't stop the hang when I hibernate with too
>> >>>>>>>> many
>> >>>>>>>> applications.
>> >>>>>>>>
>> >>>>>>>> It does stop the same hang in a different case though.
>> >>>>>>>>
>> >>>>>>>> 1. boot with init=/bin/bash
>> >>>>>>>> 2. run s2disk
>> >>>>>>>> 3. cancel the s2disk
>> >>>>>>>> 4. repeat steps 2&3
>> >>>>>>>>
>> >>>>>>>> With the patch, I can run 10s of iterations, with no hang.
>> >>>>>>>> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as
>> >>>>>>>> always).
>> >>>>>>>>
>> >>>>>>>> That's what happens on 2.6.33-rc7. On 2.6.30, there is no
>> >>>>>>>> problem.
>> >>>>>>>> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an
>> >>>>>>>> allocation
>> >>>>>>>> failure after a couple of iterations ("kthreadd: page allocation
>> >>>>>>>> failure. order:1, mode:0xd0"). It looks like it might be the
>> >>>>>>>> same
>> >>>>>>>> stop_machine thread allocation failure that causes the hang.
>> >>>>>>>>
>> >>>>>>> Have you tested it alone or on top of the previous one? If you've
>> >>>>>>> tested it
>> >>>>>>> alone, please apply the appended one in addition to it and retest.
>> >>>>>>>
>> >>>>>>> Rafael
>> >>>>>>>
>> >>>>>> I did test with both patches applied together -
>> >>>>>>
>> >>>>>> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation and
>> >>>>>> resume
>> >>>>>> 2. "reducing the number of pages that we're going to keep
>> >>>>>> preallocated
>> >>>>>> by
>> >>>>>> 20%"
>> >>>>>>
>> >>>>> In that case you can try to reduce the number of preallocated pages
>> >>>>> even
>> >>>>> more,
>> >>>>> ie. change "/ 5" to "/ 2" (for example) in the second patch.
>> >>>>>
>> >>>> It still hangs if I try to hibernate a couple of times with too many
>> >>>> applications.
>> >>>>
>> >>> Hmm. I guess I asked that before, but is this a 32-bit or 64-bit
>> >>> system and
>> >>> how much RAM is there in the box?
>> >>>
>> >>> Rafael
>> >>>
>> >> EeePC 701. 32 bit. 512Mb RAM. 350Mb swap file, on a "first-gen" SSD.
>> >>
>> >
>> > Hmm. I'd try to make free_unnecessary_pages() free all of the
>> > preallocated
>> > pages and see what happens.
>> >
>>
>> It still hangs in hibernation_snapshot() / disable_nonboot_cpus().
>> After apparently freeing over 400Mb / 100,000 pages of preallocated ram.
>>
>>
>>
>> There is a change which I missed before. When I applied your first
>> patch ("Force GFP_NOIO during suspend" etc.), it did change the hung
>> task backtraces a bit. I don't know if it tells us anything.
>>
>> Without the patch, there were two backtraces. The first backtrace
>> suggested a problem allocating pages for a kernel thread (at
>> copy_process() / try_to_free_pages()). The second showed that this
>> problem was blocking s2disk (at hibernation_snapshot() /
>> disable_nonboot_cpus() / stop_machine_create()).
>>
>> With the GFP_NOIO patch, I see only the s2disk backtrace.
>
> Can you please post this backtrace?
Sure. It's rather like the one I posted before, except
a) it only shows the one hung task (s2disk)
b) this time I had lockdep enabled
c) this time most of the lines don't have question marks.
Kernel verson:
- mainline v2.6.33-rc8-164-gaea187c, with the one patch "Force GFP_NOIO..."
Image:
http://picasaweb.google.com/lh/photo/f9KRZT2l9wCmVt-Jdggd9g?feat=directlink
INFO: task s2disk:1916 blocked for more than 120 seconds
...
Call Trace:
? _raw_spin_unlock
schedule_timeout+0x22 (timer.c:1366)
? mark_held_locks
? _raw_spin_unlock_irq
? trace_hardirqs_on_caller
? trace_hardirqs_on
wait_for_common+0xb8 (sched.c:5844)
? default_wake_function
wait_for_completion+0x12 (sched.c:5879)
kthread_create+0x75 (kthread.c:133)
? worker_thread+0x0
create_workqueue_thread+0x38 (workqueue.c:921)
? worker_thread+0x0
__create_workqueue_key+0x156 (workqueue.c:1006)
stop_machine_create+0x32 (stop_machine.c:121)
disable_nonboot_cpus+0xe (cpu.c:370)
hibernation_snapshot+0x94 (hibernate.c:266)
snapshot_ioctl+0x21b (user.c:256)
...
sys_ioctl+0x41 (ioctl.c:624)
Thanks
Alan
On Tuesday 23 February 2010, Alan Jenkins wrote:
> On 2/22/10, Rafael J. Wysocki <[email protected]> wrote:
> > On Monday 22 February 2010, Alan Jenkins wrote:
> >> Rafael J. Wysocki wrote:
> >> > On Friday 19 February 2010, Alan Jenkins wrote:
> >> >
> >> >> On 2/18/10, Rafael J. Wysocki <[email protected]> wrote:
> >> >>
> >> >>> On Thursday 18 February 2010, Alan Jenkins wrote:
> >> >>>
> >> >>>> On 2/17/10, Rafael J. Wysocki <[email protected]> wrote:
> >> >>>>
> >> >>>>> On Wednesday 17 February 2010, Alan Jenkins wrote:
> >> >>>>>
> >> >>>>>> On 2/16/10, Rafael J. Wysocki <[email protected]> wrote:
> >> >>>>>>
> >> >>>>>>> On Tuesday 16 February 2010, Alan Jenkins wrote:
> >> >>>>>>>
> >> >>>>>>>> On 2/16/10, Alan Jenkins <[email protected]> wrote:
> >> >>>>>>>>
> >> >>>>>>>>> On 2/15/10, Rafael J. Wysocki <[email protected]> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>>> On Tuesday 09 February 2010, Alan Jenkins wrote:
> >> >>>>>>>>>>
> >> >>>>>>>>>>> Perhaps I spoke too soon. I see the same hang if I run too
> >> >>>>>>>>>>> many
> >> >>>>>>>>>>> applications. The first hibernation fails with "not enough
> >> >>>>>>>>>>> swap"
> >> >>>>>>>>>>> as
> >> >>>>>>>>>>> expected, but the second or third attempt hangs (with the same
> >> >>>>>>>>>>> backtrace
> >> >>>>>>>>>>> as before).
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> The patch definitely helps though. Without the patch, I see a
> >> >>>>>>>>>>> hang
> >> >>>>>>>>>>> the
> >> >>>>>>>>>>> first time I try to hibernate with too many applications
> >> >>>>>>>>>>> running.
> >> >>>>>>>>>>>
> >> >>>>>>>>>> Well, I have an idea.
> >> >>>>>>>>>>
> >> >>>>>>>>>> Can you try to apply the appended patch in addition and see if
> >> >>>>>>>>>> that
> >> >>>>>>>>>> helps?
> >> >>>>>>>>>>
> >> >>>>>>>>>> Rafael
> >> >>>>>>>>>>
> >> >>>>>>>>> It doesn't seem to help.
> >> >>>>>>>>>
> >> >>>>>>>> To be clear: It doesn't stop the hang when I hibernate with too
> >> >>>>>>>> many
> >> >>>>>>>> applications.
> >> >>>>>>>>
> >> >>>>>>>> It does stop the same hang in a different case though.
> >> >>>>>>>>
> >> >>>>>>>> 1. boot with init=/bin/bash
> >> >>>>>>>> 2. run s2disk
> >> >>>>>>>> 3. cancel the s2disk
> >> >>>>>>>> 4. repeat steps 2&3
> >> >>>>>>>>
> >> >>>>>>>> With the patch, I can run 10s of iterations, with no hang.
> >> >>>>>>>> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as
> >> >>>>>>>> always).
> >> >>>>>>>>
> >> >>>>>>>> That's what happens on 2.6.33-rc7. On 2.6.30, there is no
> >> >>>>>>>> problem.
> >> >>>>>>>> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an
> >> >>>>>>>> allocation
> >> >>>>>>>> failure after a couple of iterations ("kthreadd: page allocation
> >> >>>>>>>> failure. order:1, mode:0xd0"). It looks like it might be the
> >> >>>>>>>> same
> >> >>>>>>>> stop_machine thread allocation failure that causes the hang.
> >> >>>>>>>>
> >> >>>>>>> Have you tested it alone or on top of the previous one? If you've
> >> >>>>>>> tested it
> >> >>>>>>> alone, please apply the appended one in addition to it and retest.
> >> >>>>>>>
> >> >>>>>>> Rafael
> >> >>>>>>>
> >> >>>>>> I did test with both patches applied together -
> >> >>>>>>
> >> >>>>>> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation and
> >> >>>>>> resume
> >> >>>>>> 2. "reducing the number of pages that we're going to keep
> >> >>>>>> preallocated
> >> >>>>>> by
> >> >>>>>> 20%"
> >> >>>>>>
> >> >>>>> In that case you can try to reduce the number of preallocated pages
> >> >>>>> even
> >> >>>>> more,
> >> >>>>> ie. change "/ 5" to "/ 2" (for example) in the second patch.
> >> >>>>>
> >> >>>> It still hangs if I try to hibernate a couple of times with too many
> >> >>>> applications.
> >> >>>>
> >> >>> Hmm. I guess I asked that before, but is this a 32-bit or 64-bit
> >> >>> system and
> >> >>> how much RAM is there in the box?
> >> >>>
> >> >>> Rafael
> >> >>>
> >> >> EeePC 701. 32 bit. 512Mb RAM. 350Mb swap file, on a "first-gen" SSD.
> >> >>
> >> >
> >> > Hmm. I'd try to make free_unnecessary_pages() free all of the
> >> > preallocated
> >> > pages and see what happens.
> >> >
> >>
> >> It still hangs in hibernation_snapshot() / disable_nonboot_cpus().
> >> After apparently freeing over 400Mb / 100,000 pages of preallocated ram.
> >>
> >>
> >>
> >> There is a change which I missed before. When I applied your first
> >> patch ("Force GFP_NOIO during suspend" etc.), it did change the hung
> >> task backtraces a bit. I don't know if it tells us anything.
> >>
> >> Without the patch, there were two backtraces. The first backtrace
> >> suggested a problem allocating pages for a kernel thread (at
> >> copy_process() / try_to_free_pages()). The second showed that this
> >> problem was blocking s2disk (at hibernation_snapshot() /
> >> disable_nonboot_cpus() / stop_machine_create()).
> >>
> >> With the GFP_NOIO patch, I see only the s2disk backtrace.
> >
> > Can you please post this backtrace?
>
> Sure. It's rather like the one I posted before, except
>
> a) it only shows the one hung task (s2disk)
> b) this time I had lockdep enabled
> c) this time most of the lines don't have question marks.
Well, it still looks like we're waiting for create_workqueue_thread() to
return, which probably is trying to allocate memory for the thread
structure.
My guess is that the preallocated memory pages freed by
free_unnecessary_pages() go into a place from where they cannot be taken for
subsequent NOIO allocations. I have no idea why that happens though.
To test that theory you can try to change GFP_IOFS to GFP_KERNEL in the
calls to clear_gfp_allowed_mask() in kernel/power/hibernate.c (and in
kernel/power/suspend.c for completness).
Rafael
On Tue, 23 Feb 2010 22:13:56 +0100
"Rafael J. Wysocki" <[email protected]> wrote:
> Well, it still looks like we're waiting for create_workqueue_thread() to
> return, which probably is trying to allocate memory for the thread
> structure.
>
> My guess is that the preallocated memory pages freed by
> free_unnecessary_pages() go into a place from where they cannot be taken for
> subsequent NOIO allocations. I have no idea why that happens though.
>
> To test that theory you can try to change GFP_IOFS to GFP_KERNEL in the
> calls to clear_gfp_allowed_mask() in kernel/power/hibernate.c (and in
> kernel/power/suspend.c for completness).
>
If allocation of kernel threads for stop_machine_run() is the problem,
What happens when
1. use CONIFG_4KSTACK
or
2. make use of stop_machine_create(), stop_machine_destroy().
A new interface added by this commit.
http://git.kernel.org/?p=linux/kernel/git/torvalds/ linux-2.6.git;a=commit;h=9ea09af3bd3090e8349ca2899ca2011bd94cda85
You can do no-fail stop_machine_run().
Thanks,
-Kame
On 2/23/10, Rafael J. Wysocki <[email protected]> wrote:
> On Tuesday 23 February 2010, Alan Jenkins wrote:
>> On 2/22/10, Rafael J. Wysocki <[email protected]> wrote:
>> > On Monday 22 February 2010, Alan Jenkins wrote:
>> >> Rafael J. Wysocki wrote:
>> >> > On Friday 19 February 2010, Alan Jenkins wrote:
>> >> >
>> >> >> On 2/18/10, Rafael J. Wysocki <[email protected]> wrote:
>> >> >>
>> >> >>> On Thursday 18 February 2010, Alan Jenkins wrote:
>> >> >>>
>> >> >>>> On 2/17/10, Rafael J. Wysocki <[email protected]> wrote:
>> >> >>>>
>> >> >>>>> On Wednesday 17 February 2010, Alan Jenkins wrote:
>> >> >>>>>
>> >> >>>>>> On 2/16/10, Rafael J. Wysocki <[email protected]> wrote:
>> >> >>>>>>
>> >> >>>>>>> On Tuesday 16 February 2010, Alan Jenkins wrote:
>> >> >>>>>>>
>> >> >>>>>>>> On 2/16/10, Alan Jenkins <[email protected]>
>> >> >>>>>>>> wrote:
>> >> >>>>>>>>
>> >> >>>>>>>>> On 2/15/10, Rafael J. Wysocki <[email protected]> wrote:
>> >> >>>>>>>>>
>> >> >>>>>>>>>> On Tuesday 09 February 2010, Alan Jenkins wrote:
>> >> >>>>>>>>>>
>> >> >>>>>>>>>>> Perhaps I spoke too soon. I see the same hang if I run too
>> >> >>>>>>>>>>> many
>> >> >>>>>>>>>>> applications. The first hibernation fails with "not enough
>> >> >>>>>>>>>>> swap"
>> >> >>>>>>>>>>> as
>> >> >>>>>>>>>>> expected, but the second or third attempt hangs (with the
>> >> >>>>>>>>>>> same
>> >> >>>>>>>>>>> backtrace
>> >> >>>>>>>>>>> as before).
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> The patch definitely helps though. Without the patch, I
>> >> >>>>>>>>>>> see a
>> >> >>>>>>>>>>> hang
>> >> >>>>>>>>>>> the
>> >> >>>>>>>>>>> first time I try to hibernate with too many applications
>> >> >>>>>>>>>>> running.
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>> Well, I have an idea.
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> Can you try to apply the appended patch in addition and see
>> >> >>>>>>>>>> if
>> >> >>>>>>>>>> that
>> >> >>>>>>>>>> helps?
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> Rafael
>> >> >>>>>>>>>>
>> >> >>>>>>>>> It doesn't seem to help.
>> >> >>>>>>>>>
>> >> >>>>>>>> To be clear: It doesn't stop the hang when I hibernate with
>> >> >>>>>>>> too
>> >> >>>>>>>> many
>> >> >>>>>>>> applications.
>> >> >>>>>>>>
>> >> >>>>>>>> It does stop the same hang in a different case though.
>> >> >>>>>>>>
>> >> >>>>>>>> 1. boot with init=/bin/bash
>> >> >>>>>>>> 2. run s2disk
>> >> >>>>>>>> 3. cancel the s2disk
>> >> >>>>>>>> 4. repeat steps 2&3
>> >> >>>>>>>>
>> >> >>>>>>>> With the patch, I can run 10s of iterations, with no hang.
>> >> >>>>>>>> Without the patch, it soon hangs, (in disable_nonboot_cpus(),
>> >> >>>>>>>> as
>> >> >>>>>>>> always).
>> >> >>>>>>>>
>> >> >>>>>>>> That's what happens on 2.6.33-rc7. On 2.6.30, there is no
>> >> >>>>>>>> problem.
>> >> >>>>>>>> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an
>> >> >>>>>>>> allocation
>> >> >>>>>>>> failure after a couple of iterations ("kthreadd: page
>> >> >>>>>>>> allocation
>> >> >>>>>>>> failure. order:1, mode:0xd0"). It looks like it might be the
>> >> >>>>>>>> same
>> >> >>>>>>>> stop_machine thread allocation failure that causes the hang.
>> >> >>>>>>>>
>> >> >>>>>>> Have you tested it alone or on top of the previous one? If
>> >> >>>>>>> you've
>> >> >>>>>>> tested it
>> >> >>>>>>> alone, please apply the appended one in addition to it and
>> >> >>>>>>> retest.
>> >> >>>>>>>
>> >> >>>>>>> Rafael
>> >> >>>>>>>
>> >> >>>>>> I did test with both patches applied together -
>> >> >>>>>>
>> >> >>>>>> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation
>> >> >>>>>> and
>> >> >>>>>> resume
>> >> >>>>>> 2. "reducing the number of pages that we're going to keep
>> >> >>>>>> preallocated
>> >> >>>>>> by
>> >> >>>>>> 20%"
>> >> >>>>>>
>> >> >>>>> In that case you can try to reduce the number of preallocated
>> >> >>>>> pages
>> >> >>>>> even
>> >> >>>>> more,
>> >> >>>>> ie. change "/ 5" to "/ 2" (for example) in the second patch.
>> >> >>>>>
>> >> >>>> It still hangs if I try to hibernate a couple of times with too
>> >> >>>> many
>> >> >>>> applications.
>> >> >>>>
>> >> >>> Hmm. I guess I asked that before, but is this a 32-bit or 64-bit
>> >> >>> system and
>> >> >>> how much RAM is there in the box?
>> >> >>>
>> >> >>> Rafael
>> >> >>>
>> >> >> EeePC 701. 32 bit. 512Mb RAM. 350Mb swap file, on a "first-gen"
>> >> >> SSD.
>> >> >>
>> >> >
>> >> > Hmm. I'd try to make free_unnecessary_pages() free all of the
>> >> > preallocated
>> >> > pages and see what happens.
>> >> >
>> >>
>> >> It still hangs in hibernation_snapshot() / disable_nonboot_cpus().
>> >> After apparently freeing over 400Mb / 100,000 pages of preallocated
>> >> ram.
>> >>
>> >>
>> >>
>> >> There is a change which I missed before. When I applied your first
>> >> patch ("Force GFP_NOIO during suspend" etc.), it did change the hung
>> >> task backtraces a bit. I don't know if it tells us anything.
>> >>
>> >> Without the patch, there were two backtraces. The first backtrace
>> >> suggested a problem allocating pages for a kernel thread (at
>> >> copy_process() / try_to_free_pages()). The second showed that this
>> >> problem was blocking s2disk (at hibernation_snapshot() /
>> >> disable_nonboot_cpus() / stop_machine_create()).
>> >>
>> >> With the GFP_NOIO patch, I see only the s2disk backtrace.
>> >
>> > Can you please post this backtrace?
>>
>> Sure. It's rather like the one I posted before, except
>>
>> a) it only shows the one hung task (s2disk)
>> b) this time I had lockdep enabled
>> c) this time most of the lines don't have question marks.
>
> Well, it still looks like we're waiting for create_workqueue_thread() to
> return, which probably is trying to allocate memory for the thread
> structure.
>
> My guess is that the preallocated memory pages freed by
> free_unnecessary_pages() go into a place from where they cannot be taken for
> subsequent NOIO allocations. I have no idea why that happens though.
>
> To test that theory you can try to change GFP_IOFS to GFP_KERNEL in the
> calls to clear_gfp_allowed_mask() in kernel/power/hibernate.c (and in
> kernel/power/suspend.c for completness).
Effectively forcing GFP_NOWAIT, so the allocation should fail instead
of hanging?
It seems to stop the hang, but I don't see any other difference - the
hibernation process isn't stopped earlier, and I don't get any new
kernel messages about allocation failures. I wonder if it's because
GFP_NOWAIT triggers ALLOC_HARDER.
I have other evidence which argues for your theory:
[ successful s2disk, with forced NOIO (but not NOWAIT), and test code
as attached ]
Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.
1280 GFP_NOWAIT allocations of order 0 are possible
640 GFP_NOWAIT allocations of order 1 are possible
320 GFP_NOWAIT allocations of order 2 are possible
[ note - 1280 pages is the maximum test allocation used here. The
test code is only accurate when talking about smaller numbers of free
pages ]
1280 GFP_KERNEL allocations of order 0 are possible
640 GFP_KERNEL allocations of order 1 are possible
320 GFP_KERNEL allocations of order 2 are possible
PM: Preallocating image memory...
212 GFP_NOWAIT allocations of order 0 are possible
102 GFP_NOWAIT allocations of order 1 are possible
50 GFP_NOWAIT allocations of order 2 are possible
Freeing all 90083 preallocated pages
(and 0 highmem pages, out of 0)
190 GFP_NOWAIT allocations of order 0 are possible
102 GFP_NOWAIT allocations of order 1 are possible
50 GFP_NOWAIT allocations of order 2 are possible
1280 GFP_KERNEL allocations of order 0 are possible
640 GFP_KERNEL allocations of order 1 are possible
320 GFP_KERNEL allocations of order 2 are possible
done (allocated 90083 pages)
It looks like you're right and the freed pages are not accessible with
GFP_NOWAIT for some reason.
I also tried a number of test runs with too many applications, and saw this:
Freeing all 104006 preallocated pages ...
65 GFP_NOWAIT allocations of order 0 ...
18 GFP_NOWAIT allocations of order 1 ...
9 GFP_NOWAIT allocations of order 2 ...
0 GFP_KERNEL allocations of order 0 are possible
...
Disabling nonboot cpus ...
...
PM: Hibernation image created
Force enabled HPET at resume
PM: early thaw of devices complete after ... msecs
<hang, no backtrace visible even after 120 seconds>
I'm not bothered by the new hang; the test code will inevitably have
some side effects. I'm not sure why GFP_KERNEL allocations would fail
in this scenario though... perhaps the difference is that we've
swapped out the entire userspace so GFP_IO doesn't help.
Regards
Alan
On 2/24/10, KAMEZAWA Hiroyuki <[email protected]> wrote:
> On Tue, 23 Feb 2010 22:13:56 +0100
> "Rafael J. Wysocki" <[email protected]> wrote:
>
>> Well, it still looks like we're waiting for create_workqueue_thread() to
>> return, which probably is trying to allocate memory for the thread
>> structure.
>>
>> My guess is that the preallocated memory pages freed by
>> free_unnecessary_pages() go into a place from where they cannot be taken
>> for
>> subsequent NOIO allocations. I have no idea why that happens though.
>>
>> To test that theory you can try to change GFP_IOFS to GFP_KERNEL in the
>> calls to clear_gfp_allowed_mask() in kernel/power/hibernate.c (and in
>> kernel/power/suspend.c for completness).
>>
>
> If allocation of kernel threads for stop_machine_run() is the problem,
>
> What happens when
> 1. use CONIFG_4KSTACK
Interesting question. 4KSTACK doesn't stop it though; it hangs in the
same place.
> or
> 2. make use of stop_machine_create(), stop_machine_destroy().
> A new interface added by this commit.
> http://git.kernel.org/?p=linux/kernel/git/torvalds/
> linux-2.6.git;a=commit;h=9ea09af3bd3090e8349ca2899ca2011bd94cda85
> You can do no-fail stop_machine_run().
>
> Thanks,
> -Kame
Since this is a uni-processor machine that would make it a single 4K
allocation. AIUI this is supposed to be ok. The hibernation code
tries to make sure there is over 1000x that much free RAM (ish), in
anticipation of this sort of requirement.
There appear to be some deficiencies in the way this allowance works,
which have recently been exposed. And unfortunately the allocation
hangs instead of failing, so we're in unclean shutdown territory.
I have three test scenarios at the moment. I've tested two patches
which appear to fix the common cases, but there's still a third test
scenario to figure out. (Repeated hibernation attempts with
insufficient swap - encountered during real-world use, believe it or
not).
Alan
On Wednesday 24 February 2010, KAMEZAWA Hiroyuki wrote:
> On Tue, 23 Feb 2010 22:13:56 +0100
> "Rafael J. Wysocki" <[email protected]> wrote:
>
> > Well, it still looks like we're waiting for create_workqueue_thread() to
> > return, which probably is trying to allocate memory for the thread
> > structure.
> >
> > My guess is that the preallocated memory pages freed by
> > free_unnecessary_pages() go into a place from where they cannot be taken for
> > subsequent NOIO allocations. I have no idea why that happens though.
> >
> > To test that theory you can try to change GFP_IOFS to GFP_KERNEL in the
> > calls to clear_gfp_allowed_mask() in kernel/power/hibernate.c (and in
> > kernel/power/suspend.c for completness).
> >
>
> If allocation of kernel threads for stop_machine_run() is the problem,
>
> What happens when
> 1. use CONIFG_4KSTACK
> or
> 2. make use of stop_machine_create(), stop_machine_destroy().
> A new interface added by this commit.
> http://git.kernel.org/?p=linux/kernel/git/torvalds/ linux-2.6.git;a=commit;h=9ea09af3bd3090e8349ca2899ca2011bd94cda85
> You can do no-fail stop_machine_run().
Well, that would probably help in this particular case, but the root cause
seems to be that the (theoretically) freed memory cannot be used for NOIO
allocations for some reason, which is shown by the Alan's testing.
Generally speaking, we use __free_page() to release some pages preallocated
for the hibernation image, but the memory subsystem refuses to use these
pages for NOIO allocations made later. However, it evidently is able to use
them is __GFP_WAIT is unset in the mask.
Is this behavior intentional?
Rafael
On Wednesday 24 February 2010, Alan Jenkins wrote:
> On 2/23/10, Rafael J. Wysocki <[email protected]> wrote:
...
> > My guess is that the preallocated memory pages freed by
> > free_unnecessary_pages() go into a place from where they cannot be taken for
> > subsequent NOIO allocations. I have no idea why that happens though.
> >
> > To test that theory you can try to change GFP_IOFS to GFP_KERNEL in the
> > calls to clear_gfp_allowed_mask() in kernel/power/hibernate.c (and in
> > kernel/power/suspend.c for completness).
>
> Effectively forcing GFP_NOWAIT, so the allocation should fail instead
> of hanging?
>
> It seems to stop the hang, but I don't see any other difference - the
> hibernation process isn't stopped earlier, and I don't get any new
> kernel messages about allocation failures. I wonder if it's because
> GFP_NOWAIT triggers ALLOC_HARDER.
>
> I have other evidence which argues for your theory:
>
> [ successful s2disk, with forced NOIO (but not NOWAIT), and test code
> as attached ]
>
> Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.
> 1280 GFP_NOWAIT allocations of order 0 are possible
> 640 GFP_NOWAIT allocations of order 1 are possible
> 320 GFP_NOWAIT allocations of order 2 are possible
>
> [ note - 1280 pages is the maximum test allocation used here. The
> test code is only accurate when talking about smaller numbers of free
> pages ]
>
> 1280 GFP_KERNEL allocations of order 0 are possible
> 640 GFP_KERNEL allocations of order 1 are possible
> 320 GFP_KERNEL allocations of order 2 are possible
>
> PM: Preallocating image memory...
> 212 GFP_NOWAIT allocations of order 0 are possible
> 102 GFP_NOWAIT allocations of order 1 are possible
> 50 GFP_NOWAIT allocations of order 2 are possible
>
> Freeing all 90083 preallocated pages
> (and 0 highmem pages, out of 0)
> 190 GFP_NOWAIT allocations of order 0 are possible
> 102 GFP_NOWAIT allocations of order 1 are possible
> 50 GFP_NOWAIT allocations of order 2 are possible
> 1280 GFP_KERNEL allocations of order 0 are possible
> 640 GFP_KERNEL allocations of order 1 are possible
> 320 GFP_KERNEL allocations of order 2 are possible
> done (allocated 90083 pages)
>
> It looks like you're right and the freed pages are not accessible with
> GFP_NOWAIT for some reason.
I'd expect this, really. There only is a limited number of pages you can
allocate with GFP_NOWAIT.
> I also tried a number of test runs with too many applications, and saw this:
>
> Freeing all 104006 preallocated pages ...
> 65 GFP_NOWAIT allocations of order 0 ...
> 18 GFP_NOWAIT allocations of order 1 ...
> 9 GFP_NOWAIT allocations of order 2 ...
> 0 GFP_KERNEL allocations of order 0 are possible
> ...
Now that's interesting. We've just freed 104006 pages and we can't allocate
any, so where did all of these freed pages go, actually?
OK, I think I see what the problem is. Quite embarassing, actually ...
Can you check if the patch below helps?
Rafael
---
kernel/power/snapshot.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Index: linux-2.6/kernel/power/snapshot.c
===================================================================
--- linux-2.6.orig/kernel/power/snapshot.c
+++ linux-2.6/kernel/power/snapshot.c
@@ -1181,7 +1181,7 @@ static void free_unnecessary_pages(void)
memory_bm_position_reset(©_bm);
- while (to_free_normal > 0 && to_free_highmem > 0) {
+ while (to_free_normal > 0 || to_free_highmem > 0) {
unsigned long pfn = memory_bm_next_pfn(©_bm);
struct page *page = pfn_to_page(pfn);
On 2/24/10, Rafael J. Wysocki <[email protected]> wrote:
> On Wednesday 24 February 2010, Alan Jenkins wrote:
>> On 2/23/10, Rafael J. Wysocki <[email protected]> wrote:
> ...
>> > My guess is that the preallocated memory pages freed by
>> > free_unnecessary_pages() go into a place from where they cannot be taken
>> > for
>> > subsequent NOIO allocations. I have no idea why that happens though.
>> >
>> > To test that theory you can try to change GFP_IOFS to GFP_KERNEL in the
>> > calls to clear_gfp_allowed_mask() in kernel/power/hibernate.c (and in
>> > kernel/power/suspend.c for completness).
>>
>> Effectively forcing GFP_NOWAIT, so the allocation should fail instead
>> of hanging?
>>
>> It seems to stop the hang, but I don't see any other difference - the
>> hibernation process isn't stopped earlier, and I don't get any new
>> kernel messages about allocation failures. I wonder if it's because
>> GFP_NOWAIT triggers ALLOC_HARDER.
>>
>> I have other evidence which argues for your theory:
>>
>> [ successful s2disk, with forced NOIO (but not NOWAIT), and test code
>> as attached ]
>>
>> Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.
>> 1280 GFP_NOWAIT allocations of order 0 are possible
>> 640 GFP_NOWAIT allocations of order 1 are possible
>> 320 GFP_NOWAIT allocations of order 2 are possible
>>
>> [ note - 1280 pages is the maximum test allocation used here. The
>> test code is only accurate when talking about smaller numbers of free
>> pages ]
>>
>> 1280 GFP_KERNEL allocations of order 0 are possible
>> 640 GFP_KERNEL allocations of order 1 are possible
>> 320 GFP_KERNEL allocations of order 2 are possible
>>
>> PM: Preallocating image memory...
>> 212 GFP_NOWAIT allocations of order 0 are possible
>> 102 GFP_NOWAIT allocations of order 1 are possible
>> 50 GFP_NOWAIT allocations of order 2 are possible
>>
>> Freeing all 90083 preallocated pages
>> (and 0 highmem pages, out of 0)
>> 190 GFP_NOWAIT allocations of order 0 are possible
>> 102 GFP_NOWAIT allocations of order 1 are possible
>> 50 GFP_NOWAIT allocations of order 2 are possible
>> 1280 GFP_KERNEL allocations of order 0 are possible
>> 640 GFP_KERNEL allocations of order 1 are possible
>> 320 GFP_KERNEL allocations of order 2 are possible
>> done (allocated 90083 pages)
>>
>> It looks like you're right and the freed pages are not accessible with
>> GFP_NOWAIT for some reason.
>
> I'd expect this, really. There only is a limited number of pages you can
> allocate with GFP_NOWAIT.
>
>> I also tried a number of test runs with too many applications, and saw
>> this:
>>
>> Freeing all 104006 preallocated pages ...
>> 65 GFP_NOWAIT allocations of order 0 ...
>> 18 GFP_NOWAIT allocations of order 1 ...
>> 9 GFP_NOWAIT allocations of order 2 ...
>> 0 GFP_KERNEL allocations of order 0 are possible
>> ...
>
> Now that's interesting. We've just freed 104006 pages and we can't allocate
> any, so where did all of these freed pages go, actually?
>
> OK, I think I see what the problem is. Quite embarassing, actually ...
>
> Can you check if the patch below helps?
>
> Rafael
> - while (to_free_normal > 0 && to_free_highmem > 0) {
> + while (to_free_normal > 0 || to_free_highmem > 0) {
Yes, that seems to do it. No more hangs so far (and I can still
reproduce the hang with too many applications if I un-apply the
patch).
I did see a non-fatal allocation failure though, so I'm still not sure
that the current implementation is strictly correct.
This is without the patch to increase "to_free_normal". If I get the
allocation failure again, should I try testing the "free 20% extra"
patch?
Many thanks
Alan
Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done.
PM: Preallocating image memory...
events/0: page allocation failure. order:0, mode:0xd0
Pid: 6, comm: events/0 Not tainted 2.6.33-rc8eeepc-00165-gecdaf98-dirty #101
Call Trace:
? printk+0xf/0x18
__alloc_pages_nodemask+0x46a/0x4dd
cache_alloc_refill+0x250/0x42d
kmem_cache_alloc+0x70/0xee
acpi_ps_alloc_op+0x4a/0x84
acpi_ps_create_scope_op+0xd/0x1c
acpi_ps_execute_method+0xed/0x29c
acpi_ns_evaluate+0x13b/0x241
acpi_evaluate_object+0x11c/0x243
? trace_hardirqs_on_caller+0x100/0x121
acpi_evaluate_integer+0x30/0x9c
acpi_thermal_get_temperature+0x2d/0x6a [thermal]
thermal_get_temp+0x1d/0x33 [thermal]
thermal_zone_device_update+0x29/0x1ce [thermal_sys]
? worker_thread+0x14b/0x250
thermal_zone_device_check+0xd/0xf [thermal_sys]
worker_thread+0x18d/0x250
? worker_thread+0x14b/0x250
? thermal_zone_device_check+0x0/0xf [thermal_sys]
? autoremove_wake_function+0x0/0x2f
? worker_thread+0x0/0x250
kthread+0x6a/0x6f
? kthread+0x0/0x6f
kernel_thread_helper+0x6/0x1a
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Normal per-cpu:
CPU 0: hi: 186, btch: 31 usd: 192
active_anon:4987 inactive_anon:5012 isolated_anon:0
active_file:34 inactive_file:86 isolated_file:0
unevictable:1042 dirty:0 writeback:0 unstable:0
free:1188 slab_reclaimable:1510 slab_unreclaimable:2338
mapped:1975 shmem:7985 pagetables:894 bounce:0
DMA free:2016kB min:88kB low:108kB high:132kB active_anon:0kB
inactive_anon:12kB active_file:8kB inactive_file:0kB unevictable:40kB
isolated(anon):0kB isolated(file):0kB present:15804kB mlocked:40kB
dirty:0kB writeback:0kB mapped:48kB shmem:0kB slab_reclaimable:8kB
slab_unreclaimable:4kB kernel_stack:0kB pagetables:8kB unstable:0kB
bounce:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? no
lowmem_reserve[]: 0 483 483 483
Normal free:2736kB min:2764kB low:3452kB high:4144kB
active_anon:19948kB inactive_anon:20036kB active_file:128kB
inactive_file:344kB unevictable:4128kB isolated(anon):0kB
isolated(file):0kB present:495300kB mlocked:4128kB dirty:0kB
writeback:0kB mapped:7852kB shmem:31940kB slab_reclaimable:6032kB
slab_unreclaimable:9348kB kernel_stack:920kB pagetables:3568kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:20672
all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 0*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB
0*2048kB 0*4096kB = 2016kB
Normal: 684*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 2736kB
9036 total pagecache pages
0 pages in swap cache
Swap cache stats: add 132152, delete 132152, find 22472/27086
Free swap = 63644kB
Total swap = 358392kB
128880 pages RAM
0 pages HighMem
3743 pages reserved
20404 pages shared
116286 pages non-shared
hda-intel: IRQ timing workaround is activated for card #0. Suggest a
bigger bdl_pos_adj.
Thermal: failed to read out thermal zone 0
done (allocated 105868 pages)
PM: Allocated 423472 kbytes in 25.48 seconds (16.61 MB/s)
atl2 0000:03:00.0: PCI INT A disabled
ata_piix 0000:00:1f.2: PCI INT B disabled
HDA Intel 0000:00:1b.0: PCI INT A disabled
PM: freeze of devices complete after 163.921 msecs
PM: late freeze of devices complete after 2.951 msecs
Disabling non-boot CPUs ...
On Thursday 25 February 2010, Alan Jenkins wrote:
> On 2/24/10, Rafael J. Wysocki <[email protected]> wrote:
> > On Wednesday 24 February 2010, Alan Jenkins wrote:
...
>
> > - while (to_free_normal > 0 && to_free_highmem > 0) {
> > + while (to_free_normal > 0 || to_free_highmem > 0) {
>
> Yes, that seems to do it. No more hangs so far (and I can still
> reproduce the hang with too many applications if I un-apply the
> patch).
OK, great. Is this with or without the NOIO-enforcing patch?
> I did see a non-fatal allocation failure though, so I'm still not sure
> that the current implementation is strictly correct.
>
> This is without the patch to increase "to_free_normal". If I get the
> allocation failure again, should I try testing the "free 20% extra"
> patch?
Either that or try to increase SPARE_PAGES. That should actually work with
the last patch applied. :-)
Rafael
On 2/25/10, Rafael J. Wysocki <[email protected]> wrote:
> On Thursday 25 February 2010, Alan Jenkins wrote:
>> On 2/24/10, Rafael J. Wysocki <[email protected]> wrote:
>> > On Wednesday 24 February 2010, Alan Jenkins wrote:
> ...
>>
>> > - while (to_free_normal > 0 && to_free_highmem > 0) {
>> > + while (to_free_normal > 0 || to_free_highmem > 0) {
>>
>> Yes, that seems to do it. No more hangs so far (and I can still
>> reproduce the hang with too many applications if I un-apply the
>> patch).
>
> OK, great. Is this with or without the NOIO-enforcing patch?
With.
>> I did see a non-fatal allocation failure though, so I'm still not sure
>> that the current implementation is strictly correct.
>>
>> This is without the patch to increase "to_free_normal". If I get the
>> allocation failure again, should I try testing the "free 20% extra"
>> patch?
>
> Either that or try to increase SPARE_PAGES. That should actually work with
> the last patch applied. :-)
>
> Rafael
<grin>, OK.