Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754784AbYCTIou (ORCPT ); Thu, 20 Mar 2008 04:44:50 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1753175AbYCTIoj (ORCPT ); Thu, 20 Mar 2008 04:44:39 -0400 Received: from fg-out-1718.google.com ([72.14.220.152]:35251 "EHLO fg-out-1718.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753071AbYCTIoh (ORCPT ); Thu, 20 Mar 2008 04:44:37 -0400 DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=message-id:date:from:user-agent:mime-version:to:cc:subject:references:in-reply-to:x-enigmail-version:content-type:content-transfer-encoding; b=KNUKvo99BzRpGlv+8UfSZDKRxEeOKYGpEeuk3nIMglV7YH0wXlnrscS1bDwla5ih7/Gk2gc4+dDIJx0nCpBSouW7QBkagkTJ8iJJSg+NUTyfD2WnSt9AR9Ohb/sbnK2t9W/qb5NBLkTHAKvjc2etxYSegDCTwAs+hY1JYvWlgmM= Message-ID: <47E223E9.6000908@gmail.com> Date: Thu, 20 Mar 2008 09:44:25 +0100 From: Jiri Slaby User-Agent: Thunderbird 2.0.0.12 (X11/20080213) MIME-Version: 1.0 To: Kevin Shanahan CC: linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: OOM (HighMem) on linux 2.6.24.2 References: <1205974089.4023.20.camel@kulgan.wumi.org.au> In-Reply-To: <1205974089.4023.20.camel@kulgan.wumi.org.au> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-2; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 6007 Lines: 87 On 03/20/2008 01:48 AM, Kevin Shanahan wrote: > I'm running a vanilla 2.6.24.2 kernel on an x86 system with 2GB physical > RAM. We had a problem yesterday where Samba and Apache both "died" and > AFAIK, we weren't able to ssh into the system. One of our staff was able > to reboot from the console though. > > So, I found the following information in the system log just before it > went down: > > Mar 19 17:01:21 hermes kernel: smbd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 > Mar 19 17:01:21 hermes kernel: Pid: 2757, comm: smbd Not tainted 2.6.24.2 #1 > Mar 19 17:01:21 hermes kernel: [] oom_kill_process+0x54/0xf8 > Mar 19 17:01:21 hermes kernel: [] out_of_memory+0x15f/0x194 > Mar 19 17:01:21 hermes kernel: [] __alloc_pages+0x239/0x2c7 > Mar 19 17:01:21 hermes kernel: [] sync_page+0x0/0x40 > Mar 19 17:01:21 hermes kernel: [] preempt_schedule+0x3a/0x55 > Mar 19 17:01:21 hermes kernel: [] __do_page_cache_readahead+0xd5/0x1be > Mar 19 17:01:21 hermes kernel: [] do_page_cache_readahead+0x49/0x53 > Mar 19 17:01:21 hermes kernel: [] filemap_fault+0x19a/0x3b4 > Mar 19 17:01:21 hermes kernel: [] wakeup_kswapd+0x2d/0x70 > Mar 19 17:01:22 hermes kernel: [] __do_fault+0x51/0x327 > Mar 19 17:01:22 hermes kernel: [] do_wp_page+0x42f/0x43b > Mar 19 17:01:22 hermes kernel: [] handle_mm_fault+0x2aa/0x5b6 > Mar 19 17:01:22 hermes kernel: [] wake_up_new_task+0x77/0x7b > Mar 19 17:01:22 hermes kernel: [] do_page_fault+0x18d/0x530 > Mar 19 17:01:22 hermes kernel: [] sys_socketcall+0xeb/0x242 > Mar 19 17:01:22 hermes kernel: [] do_fcntl+0x1f8/0x27e > Mar 19 17:01:22 hermes kernel: [] sys_clone+0x36/0x3b > Mar 19 17:01:22 hermes kernel: [] do_page_fault+0x0/0x530 > Mar 19 17:01:22 hermes kernel: [] error_code+0x72/0x78 > Mar 19 17:01:22 hermes kernel: ======================= > Mar 19 17:01:23 hermes kernel: Mem-info: > Mar 19 17:01:23 hermes kernel: DMA per-cpu: > Mar 19 17:01:23 hermes kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 > Mar 19 17:01:23 hermes kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 > Mar 19 17:01:23 hermes kernel: Normal per-cpu: > Mar 19 17:01:23 hermes kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 93 Cold: hi: 62, btch: 15 usd: 55 > Mar 19 17:01:23 hermes kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 120 Cold: hi: 62, btch: 15 usd: 55 > Mar 19 17:01:23 hermes kernel: HighMem per-cpu: > Mar 19 17:01:23 hermes kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 46 Cold: hi: 62, btch: 15 usd: 60 > Mar 19 17:01:23 hermes kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 12 Cold: hi: 62, btch: 15 usd: 61 > Mar 19 17:01:23 hermes kernel: Active:150615 inactive:339150 dirty:0 writeback:0 unstable:0 > Mar 19 17:01:23 hermes kernel: free:12180 slab:6561 mapped:78 pagetables:2862 bounce:0 > Mar 19 17:01:23 hermes kernel: DMA free:8132kB min:68kB low:84kB high:100kB active:1872kB inactive:1256kB present:16256kB pages_scanned:5836 all_unreclaimable? yes > Mar 19 17:01:23 hermes kernel: lowmem_reserve[]: 0 873 2016 2016 > Mar 19 17:01:23 hermes kernel: Normal free:40136kB min:3744kB low:4680kB high:5616kB active:400240kB inactive:398884kB present:894080kB pages_scanned:2546811 all_unreclaimable? yes > Mar 19 17:01:23 hermes kernel: lowmem_reserve[]: 0 0 9143 9143 > Mar 19 17:01:23 hermes kernel: HighMem free:452kB min:512kB low:1736kB high:2964kB active:200720kB inactive:956076kB present:1170372kB pages_scanned:3284202 all_unreclaimable? yes > Mar 19 17:01:23 hermes kernel: lowmem_reserve[]: 0 0 0 0 > Mar 19 17:01:23 hermes kernel: DMA: 49*4kB 44*8kB 40*16kB 37*32kB 22*64kB 6*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 8132kB > Mar 19 17:01:23 hermes kernel: Normal: 9168*4kB 1*8kB 0*16kB 2*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 40136kB > Mar 19 17:01:23 hermes kernel: HighMem: 11*4kB 5*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 452kB > Mar 19 17:01:23 hermes kernel: Swap cache: add 550090, delete 550089, find 1023187/1039769, race 0+22 > Mar 19 17:01:23 hermes kernel: Free swap = 0kB > Mar 19 17:01:23 hermes kernel: Total swap = 899064kB > Mar 19 17:01:23 hermes kernel: Free swap: 0kB > Mar 19 17:01:23 hermes kernel: 524272 pages of RAM > Mar 19 17:01:23 hermes kernel: 294896 pages of HIGHMEM > Mar 19 17:01:23 hermes kernel: 5542 reserved pages > Mar 19 17:01:23 hermes kernel: 55226 pages shared > Mar 19 17:01:23 hermes kernel: 1 pages swap cached > Mar 19 17:01:23 hermes kernel: 0 pages dirty > Mar 19 17:01:23 hermes kernel: 0 pages writeback > Mar 19 17:01:23 hermes kernel: 78 pages mapped > Mar 19 17:01:23 hermes kernel: 6561 pages slab > Mar 19 17:01:23 hermes kernel: 2862 pages pagetables > Mar 19 17:01:23 hermes kernel: Out of memory: kill process 5928 (apache2) score 40674 or a child > Mar 19 17:01:23 hermes kernel: Killed process 5928 (apache2) > > Just wondering if anyone could help me with interpreting what is going > on here. Does it look there's anything weird going on in kernel space > based on this, or did something in userspace really just exhaust all > memory? lsmod, lspci, lsusb, .config, /proc/mounts would be good, please, I think I get this too time to time on latest -mm (akpm). Could somebody from -mm (memory) interpret the output, I must admit, I don't understand too much by who all the memory is held. The inactive 339150 is memory which was not touched for longer time got from __get_free_page* and might be swapped out, right? -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/