Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758131AbZF2Sy6 (ORCPT ); Mon, 29 Jun 2009 14:54:58 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752304AbZF2Syv (ORCPT ); Mon, 29 Jun 2009 14:54:51 -0400 Received: from mail-gx0-f226.google.com ([209.85.217.226]:49881 "EHLO mail-gx0-f226.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750755AbZF2Syu convert rfc822-to-8bit (ORCPT ); Mon, 29 Jun 2009 14:54:50 -0400 DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=LUQQEBnY5h1/8QNcgWCnYar9NEX3j3S1dCoQ7DJSFaP3fZ74p89f3pPnUZaN7h3EQb wCCQqDz52a2xlmCPJgM0WZRydZS9ZYQ6CbIjBR4Q8ZNUjZnXhkeDjc2xByUqgec+6VAj CcE+40wn7a/BGQxyLMESejXUelJjTiUmd6CV4= MIME-Version: 1.0 In-Reply-To: <20090629095729.cc9f183c.akpm@linux-foundation.org> References: <2f11576a0906290048t29667ae0sd75c96d023b113e2@mail.gmail.com> <7561.1245768237@redhat.com> <26537.1246086769@redhat.com> <20090627125412.GA1667@cmpxchg.org> <20090628113246.GA18409@localhost> <28c262360906280630n557bb182n5079e33d21ea4a83@mail.gmail.com> <2f11576a0906280749v25ab725dn8f98fbc1d2e5a5fd@mail.gmail.com> <28c262360906280947o6f9358ddh20ab549e875282a9@mail.gmail.com> <17087.1246279435@redhat.com> <20090629095729.cc9f183c.akpm@linux-foundation.org> Date: Tue, 30 Jun 2009 03:54:51 +0900 X-Google-Sender-Auth: 09dce198fb917f72 Message-ID: <2f11576a0906291154j727165e0nebdc3813d7af3158@mail.gmail.com> Subject: Re: Found the commit that causes the OOMs From: KOSAKI Motohiro To: Andrew Morton Cc: David Howells , Minchan Kim , Wu Fengguang , Johannes Weiner , "riel@redhat.com" , LKML , Christoph Lameter , "peterz@infradead.org" , "tytso@mit.edu" , "linux-mm@kvack.org" , "elladan@eskimo.com" , "npiggin@suse.de" , "Barnes, Jesse" , KAMEZAWA Hiroyuki Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4019 Lines: 94 2009/6/30 Andrew Morton : > On Mon, 29 Jun 2009 13:43:55 +0100 David Howells wrote: > >> KOSAKI Motohiro wrote: >> >> > David, Can you please try to following patch? it was posted to LKML >> > about 1-2 week ago. >> > >> > Subject "[BUGFIX][PATCH] fix lumpy reclaim lru handiling at >> > isolate_lru_pages v2" >> >> It is already committed, but I ran a test on the latest Linus kernel anyway: >> >> msgctl11 invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0 >> msgctl11 cpuset=/ mems_allowed=0 >> Pid: 20366, comm: msgctl11 Not tainted 2.6.31-rc1-cachefs #144 >> Call Trace: >> ?[] ? oom_kill_process.clone.0+0xa9/0x245 >> ?[] ? __out_of_memory+0x12b/0x142 >> ?[] ? out_of_memory+0x6a/0x94 >> ?[] ? __alloc_pages_nodemask+0x42e/0x51d >> ?[] ? copy_process+0x95/0x114f >> ?[] ? __get_free_pages+0x12/0x4f >> ?[] ? copy_process+0xb8/0x114f >> ?[] ? handle_mm_fault+0x5dd/0x62f >> ?[] ? do_fork+0x13f/0x2ba >> ?[] ? do_page_fault+0x1f8/0x20d >> ?[] ? stub_clone+0x13/0x20 >> ?[] ? system_call_fastpath+0x16/0x1b >> Mem-Info: >> DMA per-cpu: >> CPU ? ?0: hi: ? ?0, btch: ? 1 usd: ? 0 >> CPU ? ?1: hi: ? ?0, btch: ? 1 usd: ? 0 >> DMA32 per-cpu: >> CPU ? ?0: hi: ?186, btch: ?31 usd: 159 >> CPU ? ?1: hi: ?186, btch: ?31 usd: ? 2 >> Active_anon:70477 active_file:1 inactive_anon:4514 >> ?inactive_file:7 unevictable:0 dirty:0 writeback:0 unstable:0 >> ?free:1954 slab:42078 mapped:237 pagetables:57791 bounce:0 > > ~170k pages unreclaimable and ~70k pages unaccounted for. > > This does not look like a reclaim problem? OK. we need learn testcase more. [read test program source code... ] this program makes `cat /proc/sys/kernel/msgmni` * 10 processes. At least, one process creation need one userland stack page (i.e. one anon) + one kernel stack page (i.e. one unaccount page) + one pagetable page. In my 1GB box environment, default msgmni is 11969. Oh well, the system physical ram (255744) is less than needed pages (11969 * 3). In addition, those processes call msgsnd(lrand48() % 99) 1000 times. msgsnd makes one kmalloc. it mean kernel makes tons random size slab heap and it become very fragment. Ummm, I think this test don't gurantee success on 1GB box. note: I use distro kernel (Fedora11: kernel-2.6.29+ ). >> DMA free:3932kB min:60kB low:72kB high:88kB active_anon:236kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:0kB present:15364kB pages_scanned:0 all_unreclaimable? no >> lowmem_reserve[]: 0 968 968 968 >> DMA32 free:3884kB min:3948kB low:4932kB high:5920kB active_anon:281672kB inactive_anon:18056kB active_file:0kB inactive_file:24kB unevictable:0kB present:992032kB pages_scanned:6 all_unreclaimable? no >> lowmem_reserve[]: 0 0 0 0 >> DMA: 180*4kB 36*8kB 3*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3936kB >> DMA32: 491*4kB 0*8kB 0*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3884kB >> 1808 total pagecache pages >> 0 pages in swap cache >> Swap cache stats: add 0, delete 0, find 0/0 >> Free swap ?= 0kB >> Total swap = 0kB >> 255744 pages RAM >> 5589 pages reserved >> 249340 pages shared >> 219039 pages non-shared >> Out of memory: kill process 11471 (msgctl11) score 112393 or a child >> Killed process 12318 (msgctl11) > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. ?For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: email@kvack.org > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/