Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S262792AbUJ1GDS (ORCPT ); Thu, 28 Oct 2004 02:03:18 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S262787AbUJ1GDR (ORCPT ); Thu, 28 Oct 2004 02:03:17 -0400 Received: from gort.metaparadigm.com ([203.117.131.12]:64422 "EHLO gort.metaparadigm.com") by vger.kernel.org with ESMTP id S262795AbUJ1Fz5 (ORCPT ); Thu, 28 Oct 2004 01:55:57 -0400 Message-ID: <41808A20.2030408@metaparadigm.com> Date: Thu, 28 Oct 2004 13:56:48 +0800 From: Michael Clark Organization: Metaparadigm Pte Ltd User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.3) Gecko/20041007 Debian/1.7.3-5 X-Accept-Language: en MIME-Version: 1.0 To: Linux Kernel Subject: Re: 2.6.9 page allocation failure. order:0, mode:0x20 References: <41808419.8070104@metaparadigm.com> In-Reply-To: <41808419.8070104@metaparadigm.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 59600 Lines: 2051 On 10/28/04 13:31, Michael Clark wrote: > Hi All, > > I've been some doing load testing to compare 2.4.27 and 2.6.9 > under heavy memory pressure (in eye of deploying a 2.6 kernel). > No benchmarks at the moment - my main concern is getting 2.6 > to survive the tests gracefully. > > So in my survival suite I run a 'make -j192 bzImage modules' > and the machine is 2-way P3 1GHz - 2GB RAM + 2GB swap. With my > .config this is enough to push the machine about 1.7GB into swap > (3.7GB virtual) during some points of the kernel compile test. > > Anyway 2.4.27 survives without a blip. > > In 2.6.9, the compile completes okay although I see the page > allocation failures logged below. I'm guessing they are not > critical and the process faulting on the page to swap in will > just sleep and retry again later?? > > The machine was only about 800MB into swap when they happened. > > It kept going and eventually went 2GB into swap at one point > with top reporting 0K swap free (2.4 only got to 1.7GB), i guess > at this point it was evicting file backed pages to free memory > to keep going, which it did manage to do. > > So, are the messages critical or not? (i didn't see any errors > from gcc). They all seem to be allocations in radix_tree_node_alloc > called from add_to_swap_cache. How to fix? Running the test again i'm getting them showing up from different places. Compile still completes with no errors. as: page allocation failure. order:0, mode:0x20 [] __alloc_pages+0x1c3/0x390 [] mempool_alloc+0x73/0x140 [] __get_free_pages+0x25/0x40 [] kmem_getpages+0x23/0xd0 [] cache_grow+0xaf/0x160 [] schedule+0x30e/0x620 [] cache_alloc_refill+0x182/0x230 [] kmem_cache_alloc+0x49/0x50 [] alloc_as_io_context+0x19/0x80 [] as_get_io_context+0x35/0x50 [] as_add_request+0x3e/0x220 [] __elv_add_request+0x45/0xb0 [] __make_request+0x2be/0x520 [] generic_make_request+0x106/0x190 [] bio_alloc+0x20/0x1e0 [] autoremove_wake_function+0x0/0x60 [] bio_clone+0x21/0xa0 [] make_request+0x241/0x310 [] ext3_mark_inode_dirty+0x50/0x60 [] wake_up_buffer+0x17/0x60 [] generic_make_request+0x106/0x190 [] submit_bio+0x70/0x120 [] test_set_page_writeback+0xbb/0x110 [] wake_up_page+0x17/0x50 [] swap_writepage+0x9d/0xd0 [] end_swap_bio_write+0x0/0x50 [] pageout+0xc5/0x110 [] shrink_list+0x211/0x480 [] page_referenced_anon+0x7c/0x90 [] __pagevec_release+0x28/0x40 [] shrink_cache+0x17c/0x3d0 [] shrink_zone+0xba/0xf0 [] shrink_caches+0x76/0x80 [] try_to_free_pages+0xc4/0x1b0 [] __alloc_pages+0x22b/0x390 [] do_wp_page+0x90/0x2e0 [] pte_alloc_map+0x9a/0xc0 [] zap_pmd_range+0x63/0x80 [] handle_mm_fault+0x185/0x1a0 [] do_page_fault+0x19c/0x5c0 [] __fput+0xb0/0x110 [] remove_vm_struct+0x77/0xa0 [] unmap_vma_list+0x1f/0x30 [] do_munmap+0x153/0x1b0 [] sys_munmap+0x51/0x80 [] do_page_fault+0x0/0x5c0 [] error_code+0x2d/0x38 as: page allocation failure. order:0, mode:0x20 [] __alloc_pages+0x1c3/0x390 [] mempool_alloc+0x73/0x140 [] __get_free_pages+0x25/0x40 [] kmem_getpages+0x23/0xd0 [] cache_grow+0xaf/0x160 [] schedule+0x30e/0x620 [] cache_alloc_refill+0x182/0x230 [] kmem_cache_alloc+0x49/0x50 [] alloc_as_io_context+0x19/0x80 [] as_get_io_context+0x35/0x50 [] as_add_request+0x3e/0x220 [] __elv_add_request+0x45/0xb0 [] __make_request+0x2be/0x520 [] generic_make_request+0x106/0x190 [] bio_alloc+0x20/0x1e0 [] autoremove_wake_function+0x0/0x60 [] bio_clone+0x21/0xa0 [] make_request+0x241/0x310 [] generic_make_request+0x106/0x190 [] submit_bio+0x70/0x120 [] test_set_page_writeback+0xbb/0x110 [] wake_up_page+0x17/0x50 [] swap_writepage+0x9d/0xd0 [] end_swap_bio_write+0x0/0x50 [] pageout+0xc5/0x110 [] shrink_list+0x211/0x480 [] page_referenced_anon+0x7c/0x90 [] __pagevec_release+0x28/0x40 [] shrink_cache+0x17c/0x3d0 [] shrink_zone+0xba/0xf0 [] shrink_caches+0x76/0x80 [] try_to_free_pages+0xc4/0x1b0 [] __alloc_pages+0x22b/0x390 [] do_wp_page+0x90/0x2e0 [] pte_alloc_map+0x9a/0xc0 [] zap_pmd_range+0x63/0x80 [] handle_mm_fault+0x185/0x1a0 [] do_page_fault+0x19c/0x5c0 [] __fput+0xb0/0x110 [] remove_vm_struct+0x77/0xa0 [] unmap_vma_list+0x1f/0x30 [] do_munmap+0x153/0x1b0 [] sys_munmap+0x51/0x80 [] do_page_fault+0x0/0x5c0 [] error_code+0x2d/0x38 cc1: page allocation failure. order:0, mode:0x20 [] __alloc_pages+0x1c3/0x390 [] mempool_alloc+0x73/0x140 [] __get_free_pages+0x25/0x40 [] kmem_getpages+0x23/0xd0 [] cache_grow+0xaf/0x160 [] schedule+0x30e/0x620 [] cache_alloc_refill+0x182/0x230 [] kmem_cache_alloc+0x49/0x50 [] alloc_as_io_context+0x19/0x80 [] as_get_io_context+0x35/0x50 [] as_add_request+0x3e/0x220 [] __elv_add_request+0x45/0xb0 [] __make_request+0x2be/0x520 [] generic_make_request+0x106/0x190 [] mtrr_ioctl+0x6b/0x510 [] bio_clone+0x2d/0xa0 [] make_request+0x241/0x310 [] generic_make_request+0x106/0x190 [] submit_bio+0x70/0x120 [] test_set_page_writeback+0xbb/0x110 [] wake_up_page+0x17/0x50 [] swap_writepage+0x9d/0xd0 [] end_swap_bio_write+0x0/0x50 [] pageout+0xc5/0x110 [] shrink_list+0x211/0x480 [] page_referenced_anon+0x7c/0x90 [] __pagevec_release+0x28/0x40 [] shrink_cache+0x17c/0x3d0 [] do_no_page+0x60/0x290 [] shrink_zone+0xba/0xf0 [] shrink_caches+0x76/0x80 [] try_to_free_pages+0xc4/0x1b0 [] __alloc_pages+0x22b/0x390 [] do_anonymous_page+0x95/0x180 [] do_no_page+0x60/0x290 [] handle_mm_fault+0x120/0x1a0 [] do_page_fault+0x19c/0x5c0 [] __rcu_process_callbacks+0xc0/0x110 [] finish_task_switch+0x3c/0x90 [] schedule+0x30e/0x620 [] __do_softirq+0xba/0xd0 [] do_page_fault+0x0/0x5c0 [] error_code+0x2d/0x38 make: page allocation failure. order:0, mode:0x20 [] __alloc_pages+0x1c3/0x390 [] mempool_alloc+0x73/0x140 [] __get_free_pages+0x25/0x40 [] kmem_getpages+0x23/0xd0 [] cache_grow+0xaf/0x160 [] schedule+0x30e/0x620 [] cache_alloc_refill+0x182/0x230 [] kmem_cache_alloc+0x49/0x50 [] alloc_as_io_context+0x19/0x80 [] as_get_io_context+0x35/0x50 [] as_add_request+0x3e/0x220 [] __elv_add_request+0x45/0xb0 [] __make_request+0x2be/0x520 [] generic_make_request+0x106/0x190 [] bio_alloc+0x20/0x1e0 [] autoremove_wake_function+0x0/0x60 [] bio_clone+0x21/0xa0 [] make_request+0x241/0x310 [] generic_make_request+0x106/0x190 [] submit_bio+0x70/0x120 [] test_set_page_writeback+0xbb/0x110 [] wake_up_page+0x17/0x50 [] swap_writepage+0x9d/0xd0 [] end_swap_bio_write+0x0/0x50 [] pageout+0xc5/0x110 [] shrink_list+0x211/0x480 [] page_referenced_anon+0x7c/0x90 [] __pagevec_release+0x28/0x40 [] shrink_cache+0x17c/0x3d0 [] shrink_zone+0xba/0xf0 [] shrink_caches+0x76/0x80 [] try_to_free_pages+0xc4/0x1b0 [] __alloc_pages+0x22b/0x390 [] find_or_create_page+0x8c/0xa0 [] grow_dev_page+0x31/0x130 [] __getblk_slow+0x98/0x110 [] __getblk+0x4f/0x60 [] ext3_getblk+0x93/0x270 [] ext3_find_entry+0x137/0x440 [] ext3_lookup+0x41/0xd0 [] real_lookup+0xd5/0x100 [] do_lookup+0x96/0xb0 [] link_path_walk+0x69f/0xd20 [] path_lookup+0x91/0x160 [] open_exec+0x30/0x100 [] do_execve+0x13/0x230 [] sys_execve+0x42/0x80 [] syscall_call+0x7/0xb make: page allocation failure. order:0, mode:0x20 [] __alloc_pages+0x1c3/0x390 [] mempool_alloc+0x73/0x140 [] __get_free_pages+0x25/0x40 [] kmem_getpages+0x23/0xd0 [] cache_grow+0xaf/0x160 [] schedule+0x30e/0x620 [] cache_alloc_refill+0x182/0x230 [] kmem_cache_alloc+0x49/0x50 [] alloc_as_io_context+0x19/0x80 [] as_get_io_context+0x35/0x50 [] as_add_request+0x3e/0x220 [] __elv_add_request+0x45/0xb0 [] __make_request+0x2be/0x520 [] generic_make_request+0x106/0x190 [] bio_alloc+0x20/0x1e0 [] autoremove_wake_function+0x0/0x60 [] bio_clone+0x21/0xa0 [] make_request+0x241/0x310 [] generic_make_request+0x106/0x190 [] submit_bio+0x70/0x120 [] test_set_page_writeback+0xbb/0x110 [] wake_up_page+0x17/0x50 [] swap_writepage+0x9d/0xd0 [] end_swap_bio_write+0x0/0x50 [] pageout+0xc5/0x110 [] shrink_list+0x211/0x480 [] page_referenced_anon+0x7c/0x90 [] __pagevec_release+0x28/0x40 [] shrink_cache+0x17c/0x3d0 [] __make_request+0x2e3/0x520 [] shrink_zone+0xba/0xf0 [] shrink_caches+0x76/0x80 [] try_to_free_pages+0xc4/0x1b0 [] __alloc_pages+0x22b/0x390 [] __get_free_pages+0x25/0x40 [] kmem_getpages+0x23/0xd0 [] cache_grow+0xaf/0x160 [] cache_alloc_refill+0x182/0x230 [] __find_get_block+0x61/0xd0 [] __kmalloc+0x88/0xa0 [] __jbd_kmalloc+0x28/0x30 [] do_get_write_access+0x232/0x670 [] __getblk+0x2b/0x60 [] ext3_get_inode_loc+0x5a/0x260 [] journal_get_write_access+0x37/0x50 [] ext3_reserve_inode_write+0x55/0xd0 [] task_rq_lock+0x38/0x70 [] ext3_mark_inode_dirty+0x2b/0x60 [] __d_lookup+0x10d/0x130 [] ext3_dirty_inode+0x8c/0x90 [] ext3_dirty_inode+0x0/0x90 [] __mark_inode_dirty+0x1da/0x1e0 [] d_lookup+0x34/0x60 [] update_atime+0xd9/0xe0 [] link_path_walk+0x769/0xd20 [] path_lookup+0x91/0x160 [] open_exec+0x30/0x100 [] do_execve+0x13/0x230 [] sys_execve+0x42/0x80 [] syscall_call+0x7/0xb > The kerenl is plain 2.6.9 + Nick Piggin's > vm-pages_scanned-active_list.patch > (as in earlier runs I saw kswapd periodically chewing CPU so I > picked this patch out of a mail from Nick in reply to someone > with a similar problem). > > BTW - 2.6 is much more responsive than 2.4 while this is all > going on - i'm just worried about these messages. > > ~mc > > cc1: page allocation failure. order:0, mode:0x20 > [] __alloc_pages+0x1c3/0x390 > [] __get_free_pages+0x25/0x40 > [] kmem_getpages+0x23/0xd0 > [] cache_grow+0xaf/0x160 > [] cache_alloc_refill+0x182/0x230 > [] kmem_cache_alloc+0x49/0x50 > [] radix_tree_node_alloc+0x1f/0x70 > [] radix_tree_insert+0xed/0x110 > [] __add_to_swap_cache+0x71/0x100 > [] add_to_swap+0x5f/0xc0 > [] shrink_list+0x442/0x480 > [] page_referenced_anon+0x7c/0x90 > [] __pagevec_release+0x28/0x40 > [] shrink_cache+0x17c/0x3d0 > [] shrink_zone+0xba/0xf0 > [] shrink_caches+0x76/0x80 > [] try_to_free_pages+0xc4/0x1b0 > [] __alloc_pages+0x22b/0x390 > [] do_anonymous_page+0x95/0x180 > [] swap_free+0x2f/0x50 > [] do_no_page+0x60/0x290 > [] handle_mm_fault+0x120/0x1a0 > [] do_page_fault+0x19c/0x5c0 > [] update_wall_time+0x16/0x40 > [] do_timer+0xc0/0xd0 > [] timer_interrupt+0xc0/0x130 > [] handle_IRQ_event+0x34/0x70 > [] do_IRQ+0xca/0x130 > [] do_page_fault+0x0/0x5c0 > [] error_code+0x2d/0x38 > uml_switch: page allocation failure. order:0, mode:0x20 > [] __alloc_pages+0x1c3/0x390 > [] __get_free_pages+0x25/0x40 > [] kmem_getpages+0x23/0xd0 > [] cache_grow+0xaf/0x160 > [] cache_alloc_refill+0x182/0x230 > [] kmem_cache_alloc+0x49/0x50 > [] radix_tree_node_alloc+0x1f/0x70 > [] radix_tree_insert+0xed/0x110 > [] __add_to_swap_cache+0x71/0x100 > [] add_to_swap+0x5f/0xc0 > [] shrink_list+0x442/0x480 > [] page_referenced_anon+0x7c/0x90 > [] __pagevec_release+0x28/0x40 > [] shrink_cache+0x17c/0x3d0 > [] shrink_zone+0xba/0xf0 > [] shrink_caches+0x76/0x80 > [] try_to_free_pages+0xc4/0x1b0 > [] __alloc_pages+0x22b/0x390 > [] __get_free_pages+0x25/0x40 > [] __pollwait+0x8c/0xd0 > [] unix_poll+0x2b/0xb0 [unix] > [] sock_poll+0x29/0x40 > [] do_pollfd+0x95/0xa0 > [] do_poll+0x6a/0xd0 > [] sys_poll+0x161/0x240 > [] __pollwait+0x0/0xd0 > [] syscall_call+0x7/0xb > cc1: page allocation failure. order:0, mode:0x20 > [] __alloc_pages+0x1c3/0x390 > [] __get_free_pages+0x25/0x40 > [] kmem_getpages+0x23/0xd0 > [] cache_grow+0xaf/0x160 > [] cache_alloc_refill+0x182/0x230 > [] kmem_cache_alloc+0x49/0x50 > [] radix_tree_node_alloc+0x1f/0x70 > [] radix_tree_insert+0xed/0x110 > [] __add_to_swap_cache+0x71/0x100 > [] add_to_swap+0x5f/0xc0 > [] shrink_list+0x442/0x480 > [] page_referenced_anon+0x7c/0x90 > [] __pagevec_release+0x28/0x40 > [] shrink_cache+0x17c/0x3d0 > [] shrink_zone+0xba/0xf0 > [] shrink_caches+0x76/0x80 > [] try_to_free_pages+0xc4/0x1b0 > [] __alloc_pages+0x22b/0x390 > [] do_anonymous_page+0x95/0x180 > [] do_no_page+0x60/0x290 > [] handle_mm_fault+0x120/0x1a0 > [] do_page_fault+0x19c/0x5c0 > [] i8042_interrupt+0x50/0x190 > [] run_timer_softirq+0xff/0x1a0 > [] scheduler_tick+0x123/0x480 > [] __do_softirq+0xba/0xd0 > [] do_page_fault+0x0/0x5c0 > [] error_code+0x2d/0x38 > uml_switch: page allocation failure. order:0, mode:0x20 > [] __alloc_pages+0x1c3/0x390 > [] __get_free_pages+0x25/0x40 > [] kmem_getpages+0x23/0xd0 > [] cache_grow+0xaf/0x160 > [] cache_alloc_refill+0x182/0x230 > [] kmem_cache_alloc+0x49/0x50 > [] radix_tree_node_alloc+0x1f/0x70 > [] radix_tree_insert+0xed/0x110 > [] __add_to_swap_cache+0x71/0x100 > [] add_to_swap+0x5f/0xc0 > [] shrink_list+0x442/0x480 > [] page_referenced_anon+0x7c/0x90 > [] __pagevec_release+0x28/0x40 > [] shrink_cache+0x17c/0x3d0 > [] shrink_zone+0xba/0xf0 > [] shrink_caches+0x76/0x80 > [] try_to_free_pages+0xc4/0x1b0 > [] __alloc_pages+0x22b/0x390 > [] __get_free_pages+0x25/0x40 > [] __pollwait+0x8c/0xd0 > [] unix_poll+0x2b/0xb0 [unix] > [] sock_poll+0x29/0x40 > [] do_pollfd+0x95/0xa0 > [] do_poll+0x6a/0xd0 > [] sys_poll+0x161/0x240 > [] __pollwait+0x0/0xd0 > [] syscall_call+0x7/0xb > cc1: page allocation failure. order:0, mode:0x20 > [] __alloc_pages+0x1c3/0x390 > [] __get_free_pages+0x25/0x40 > [] kmem_getpages+0x23/0xd0 > [] cache_grow+0xaf/0x160 > [] cache_alloc_refill+0x182/0x230 > [] radix_tree_delete+0x141/0x190 > [] kmem_cache_alloc+0x49/0x50 > [] radix_tree_node_alloc+0x1f/0x70 > [] radix_tree_insert+0xed/0x110 > [] __add_to_swap_cache+0x71/0x100 > [] add_to_swap+0x5f/0xc0 > [] shrink_list+0x442/0x480 > [] page_referenced_anon+0x7c/0x90 > [] __pagevec_release+0x28/0x40 > [] shrink_cache+0x17c/0x3d0 > [] shrink_zone+0xba/0xf0 > [] shrink_caches+0x76/0x80 > [] try_to_free_pages+0xc4/0x1b0 > [] __alloc_pages+0x22b/0x390 > [] read_swap_cache_async+0xb1/0xd0 > [] swapin_readahead+0x4f/0x80 > [] do_swap_page+0x1e4/0x290 > [] pte_alloc_map+0x9a/0xc0 > [] handle_mm_fault+0xf7/0x1a0 > [] do_page_fault+0x19c/0x5c0 > [] sd_rw_intr+0x5a/0x1e0 > [] scsi_device_unbusy+0x44/0x80 > [] run_timer_softirq+0xff/0x1a0 > [] scheduler_tick+0x123/0x480 > [] __do_softirq+0xba/0xd0 > [] do_page_fault+0x0/0x5c0 > [] error_code+0x2d/0x38 > cc1: page allocation failure. order:0, mode:0x20 > [] __alloc_pages+0x1c3/0x390 > [] __get_free_pages+0x25/0x40 > [] kmem_getpages+0x23/0xd0 > [] cache_grow+0xaf/0x160 > [] cache_alloc_refill+0x182/0x230 > [] kmem_cache_alloc+0x49/0x50 > [] radix_tree_node_alloc+0x1f/0x70 > [