Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760841AbZF2QHc (ORCPT ); Mon, 29 Jun 2009 12:07:32 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1753741AbZF2QHZ (ORCPT ); Mon, 29 Jun 2009 12:07:25 -0400 Received: from gir.skynet.ie ([193.1.99.77]:44520 "EHLO gir.skynet.ie" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753061AbZF2QHY (ORCPT ); Mon, 29 Jun 2009 12:07:24 -0400 Date: Mon, 29 Jun 2009 17:07:25 +0100 From: Mel Gorman To: Minchan Kim Cc: David Howells , Wu Fengguang , KOSAKI Motohiro , Johannes Weiner , "riel@redhat.com" , Andrew Morton , LKML , Christoph Lameter , "peterz@infradead.org" , "tytso@mit.edu" , "linux-mm@kvack.org" , "elladan@eskimo.com" , "npiggin@suse.de" , "Barnes, Jesse" Subject: Re: Found the commit that causes the OOMs Message-ID: <20090629160725.GF5065@csn.ul.ie> References: <28c262360906280630n557bb182n5079e33d21ea4a83@mail.gmail.com> <28c262360906280636l93130ffk14086314e2a6dcb7@mail.gmail.com> <20090628142239.GA20986@localhost> <2f11576a0906280801w417d1b9fpe10585b7a641d41b@mail.gmail.com> <20090628151026.GB25076@localhost> <20090629091741.ab815ae7.minchan.kim@barrios-desktop> <17678.1246270219@redhat.com> <20090629125549.GA22932@localhost> <29432.1246285300@redhat.com> <28c262360906290800v37f91d7av3642b1ad8b5f0477@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <28c262360906290800v37f91d7av3642b1ad8b5f0477@mail.gmail.com> User-Agent: Mutt/1.5.17+20080114 (2008-01-14) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4359 Lines: 97 On Tue, Jun 30, 2009 at 12:00:26AM +0900, Minchan Kim wrote: > On Mon, Jun 29, 2009 at 11:21 PM, David Howells wrote: > > Wu Fengguang wrote: > > > >> Sorry! This one compiles OK: > > > > Sadly that doesn't seem to work either: > > > > msgctl11 invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0 > > msgctl11 cpuset=/ mems_allowed=0 > > Pid: 30858, comm: msgctl11 Not tainted 2.6.31-rc1-cachefs #146 > > Call Trace: > > ?[] ? oom_kill_process.clone.0+0xa9/0x245 > > ?[] ? __out_of_memory+0x12b/0x142 > > ?[] ? out_of_memory+0x6a/0x94 > > ?[] ? __alloc_pages_nodemask+0x42e/0x51d > > ?[] ? do_wp_page+0x2c6/0x5f5 > > ?[] ? handle_mm_fault+0x5dd/0x62f > > ?[] ? do_page_fault+0x1f8/0x20d > > ?[] ? page_fault+0x1f/0x30 > > Mem-Info: > > DMA per-cpu: > > CPU ? ?0: hi: ? ?0, btch: ? 1 usd: ? 0 > > CPU ? ?1: hi: ? ?0, btch: ? 1 usd: ? 0 > > DMA32 per-cpu: > > CPU ? ?0: hi: ?186, btch: ?31 usd: ?38 > > CPU ? ?1: hi: ?186, btch: ?31 usd: 106 > > Active_anon:75040 active_file:0 inactive_anon:2031 > > ?inactive_file:0 unevictable:0 dirty:0 writeback:0 unstable:0 > > ?free:1951 slab:41499 mapped:301 pagetables:60674 bounce:0 > > DMA free:3932kB min:60kB low:72kB high:88kB active_anon:2868kB inactive_anon:384kB active_file:0kB inactive_file:0kB unevictable:0kB present:15364kB pages_scanned:0 all_unreclaimable? no > > lowmem_reserve[]: 0 968 968 968 > > DMA32 free:3872kB min:3948kB low:4932kB high:5920kB active_anon:297292kB inactive_anon:7740kB active_file:0kB inactive_file:0kB unevictable:0kB present:992032kB pages_scanned:0 all_unreclaimable? no > > lowmem_reserve[]: 0 0 0 0 > > DMA: 7*4kB 0*8kB 0*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3932kB > > DMA32: 500*4kB 2*8kB 0*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3872kB > > 1928 total pagecache pages > > 0 pages in swap cache > > Swap cache stats: add 0, delete 0, find 0/0 > > Free swap ?= 0kB > > Total swap = 0kB > > 255744 pages RAM > > 5589 pages reserved > > 238251 pages shared > > 216210 pages non-shared > > Out of memory: kill process 25221 (msgctl11) score 130560 or a child > > Killed process 26379 (msgctl11) > > Totally, I can't understand this situation. > Now, this page allocation is order zero and It is just likely GFP_HIGHUSER. > So it's unlikely interrupt context. The GFP flags that are set are #define __GFP_HIGHMEM (0x02) #define __GFP_MOVABLE (0x08) /* Page is movable */ #define __GFP_WAIT (0x10) /* Can wait and reschedule? */ #define __GFP_IO (0x40) /* Can start physical IO? */ #define __GFP_FS (0x80) /* Can call down to low-level FS? */ #define __GFP_HARDWALL (0x20000) /* Enforce hardwall cpuset memory allocs */ which are fairly permissive in terms of what action can be taken. > Buddy already has enough fallback DMA32, I think. It doesn't really. We are below the minimum watermark. It wouldn't be able to grant the allocation until a few pages had been freed. > Why kernel can't allocate page for order 0 ? > Is it allocator bug ? > If it is, it is not because the allocation failed as the watermarks were not being met. For this situation to be occuring, it has to be scanning the LRU lists and making no forward progress. Odd things to note; o active_anon is very large in comparison to inactive_anon. Is this because there is no swap and they are no longer being rotated? o Slab and pagetables are very large. Is slab genuinely unshrinkable? I think this system might be genuinely OOM. It can't reclaim memory and we are below the minimum watermarks. Is it possible there are pages that are counted as active_anon that in fact are reclaimable because they are on the wrong LRU list? If that was the case, the lack of rotation to inactive list would prevent them getting discovered. -- Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/