Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755965AbZGGGMf (ORCPT ); Tue, 7 Jul 2009 02:12:35 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1751453AbZGGGM1 (ORCPT ); Tue, 7 Jul 2009 02:12:27 -0400 Received: from mga03.intel.com ([143.182.124.21]:3434 "EHLO mga03.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751348AbZGGGM0 (ORCPT ); Tue, 7 Jul 2009 02:12:26 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.42,361,1243839600"; d="scan'208";a="162190850" Date: Tue, 7 Jul 2009 14:12:13 +0800 From: Wu Fengguang To: Gene Heskett Cc: Linux Kernel list , linux-mm@kvack.org, Minchan Kim , "Rafael J. Wysocki" , Kernel Testers List , David Howells , KOSAKI Motohiro Subject: Re: OOM killer in 2.6.31-rc2 Message-ID: <20090707061213.GA21004@localhost> References: <200907061056.00229.gene.heskett@verizon.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200907061056.00229.gene.heskett@verizon.net> User-Agent: Mutt/1.5.18 (2008-05-17) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 17152 Lines: 270 On Mon, Jul 06, 2009 at 10:56:00AM -0400, Gene Heskett wrote: > Greetings all; > > I had to hard reset this box just now as there was no response to a ctl-alt- > bksp when X was un-responsive this morning. > > I had built a 2nd version of the 2.6.31-rc2 kernel last night when I found the > video stuff appeared to have been moved in the .config and my pcHDTV-3000 > cards modules were not being built, but are now. That kernel was installed, > and if at some time in the night a module was needed, it would have been > available, but I can't make a solid connection. > > This machine will always be marked as 'tainted' because any bios update that > fixes the very early boot time oops, also leaves me with a machine that will > crash hard in 30 seconds to 3 or 4 hours. The fixes done by the oops make it > generally dead stable for weeks. That oops: > > Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-provided physical RAM map: > Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009f000 (usable) > Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved) > Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) > Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 0000000000100000 - 00000000dfee0000 (usable) > Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 00000000dfee0000 - 00000000dfee3000 (ACPI NVS) > Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 00000000dfee3000 - 00000000dfef0000 (ACPI data) > Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 00000000dfef0000 - 00000000dff00000 (reserved) > Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 00000000f0000000 - 00000000f4000000 (reserved) > Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved) > Jul 6 10:03:58 coyote kernel: [ 0.000000] BIOS-e820: 0000000100000000 - 0000000120000000 (usable) > Jul 6 10:03:58 coyote kernel: [ 0.000000] DMI 2.4 present. > Jul 6 10:03:58 coyote kernel: [ 0.000000] Phoenix BIOS detected: BIOS may corrupt low RAM, working around it. > Jul 6 10:03:58 coyote kernel: [ 0.000000] last_pfn = 0x120000 max_arch_pfn = 0x1000000 > Jul 6 10:03:58 coyote kernel: [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 > Jul 6 10:03:58 coyote kernel: [ 0.000000] ------------[ cut here ]------------ > Jul 6 10:03:58 coyote kernel: [ 0.000000] WARNING: at arch/x86/kernel/cpu/mtrr/generic.c:456 > generic_get_mtrr+0x12c/0x150() > Jul 6 10:03:58 coyote kernel: [ 0.000000] Hardware name: System Product Name > Jul 6 10:03:58 coyote kernel: [ 0.000000] mtrr: your BIOS has set up an incorrect mask, fixing it up. > Jul 6 10:03:58 coyote kernel: [ 0.000000] Modules linked in: > Jul 6 10:03:58 coyote kernel: [ 0.000000] Pid: 0, comm: swapper Not tainted 2.6.31-rc2 #2 > Jul 6 10:03:58 coyote kernel: [ 0.000000] Call Trace: > Jul 6 10:03:58 coyote kernel: [ 0.000000] [] ? generic_get_mtrr+0x12c/0x150 > Jul 6 10:03:58 coyote kernel: [ 0.000000] [] warn_slowpath_common+0x7d/0xe0 > Jul 6 10:03:58 coyote kernel: [ 0.000000] [] ? generic_get_mtrr+0x12c/0x150 > Jul 6 10:03:58 coyote kernel: [ 0.000000] [] warn_slowpath_fmt+0x33/0x50 > Jul 6 10:03:58 coyote kernel: [ 0.000000] [] generic_get_mtrr+0x12c/0x150 > Jul 6 10:03:58 coyote kernel: [ 0.000000] [] mtrr_trim_uncached_memory+0x85/0x368 > Jul 6 10:03:58 coyote kernel: [ 0.000000] [] ? mtrr_bp_init+0x1d9/0x2bb > Jul 6 10:03:58 coyote kernel: [ 0.000000] [] setup_arch+0x52c/0xa33 > Jul 6 10:03:58 coyote kernel: [ 0.000000] [] ? thermal_get_trip_type+0x0/0x9c > Jul 6 10:03:58 coyote kernel: [ 0.000000] [] start_kernel+0xb2/0x38b > Jul 6 10:03:58 coyote kernel: [ 0.000000] [] i386_start_kernel+0x84/0xb0 > Jul 6 10:03:58 coyote kernel: [ 0.000000] ---[ end trace a7919e7f17c0a725 ]--- > Jul 6 10:03:58 coyote kernel: [ 0.000000] Scanning 0 areas for low memory corruption > Jul 6 10:03:58 coyote kernel: [ 0.000000] modified physical RAM map: > Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 0000000000000000 - 0000000000010000 (reserved) > Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 0000000000010000 - 000000000009f000 (usable) > Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 000000000009f000 - 00000000000a0000 (reserved) > Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 00000000000f0000 - 0000000000100000 (reserved) > Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 0000000000100000 - 00000000dfee0000 (usable) > Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 00000000dfee0000 - 00000000dfee3000 (ACPI NVS) > Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 00000000dfee3000 - 00000000dfef0000 (ACPI data) > Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 00000000dfef0000 - 00000000dff00000 (reserved) > Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 00000000f0000000 - 00000000f4000000 (reserved) > Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 00000000fec00000 - 0000000100000000 (reserved) > Jul 6 10:03:58 coyote kernel: [ 0.000000] modified: 0000000100000000 - 0000000120000000 (usable) > Jul 6 10:03:58 coyote kernel: [ 0.000000] init_memory_mapping: 0000000000000000-00000000379fe000 > Jul 6 10:03:58 coyote kernel: [ 0.000000] NX (Execute Disable) protection: active > > You all have seen this one before, several times. I have asked that > since its a good fix, that the kernel not be marked tainted in that > instance. I would run the asus bios that didn't do that _IF_ it was > stable. 2 newer versions are _not_ stable, this is stable after the fix. > > The machine has 4G of ram & is I believe 'pae' > > The oom's first stanza: > > Jul 6 06:45:01 coyote kernel: [78748.106803] X invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0 > Jul 6 06:45:01 coyote kernel: [78748.106808] Pid: 3068, comm: X Tainted: G W 2.6.31-rc2 #1 > Jul 6 06:45:01 coyote kernel: [78748.106811] Call Trace: > Jul 6 06:45:01 coyote kernel: [78748.106818] [] ? printk+0x23/0x40 > Jul 6 06:45:01 coyote kernel: [78748.106823] [] oom_kill_process+0x178/0x270 > Jul 6 06:45:01 coyote kernel: [78748.106827] [] ? badness+0x14d/0x220 > Jul 6 06:45:01 coyote kernel: [78748.106830] [] __out_of_memory+0x142/0x170 > Jul 6 06:45:01 coyote kernel: [78748.106834] [] out_of_memory+0x59/0xc0 > Jul 6 06:45:01 coyote kernel: [78748.106837] [] __alloc_pages_nodemask+0x4f7/0x510 > Jul 6 06:45:01 coyote kernel: [78748.106841] [] __get_free_pages+0x23/0x50 > Jul 6 06:45:01 coyote kernel: [78748.106845] [] __pollwait+0xb2/0xf0 > Jul 6 06:45:01 coyote kernel: [78748.106848] [] unix_poll+0x28/0xc0 > Jul 6 06:45:01 coyote kernel: [78748.106851] [] sock_poll+0x1e/0x40 > Jul 6 06:45:01 coyote kernel: [78748.106853] [] do_select+0x34e/0x6b0 > Jul 6 06:45:01 coyote kernel: [78748.106871] [] ? __pollwait+0x0/0xf0 > Jul 6 06:45:01 coyote kernel: [78748.106874] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106877] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106879] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106882] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106884] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106887] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106890] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106892] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106895] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106897] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106900] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106902] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106905] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106907] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106910] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106913] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106915] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106918] [] ? pollwake+0x0/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106920] [] core_sys_select+0x212/0x350 > Jul 6 06:45:01 coyote kernel: [78748.106925] [] ? poll_select_set_timeout+0x82/0x90 > Jul 6 06:45:01 coyote kernel: [78748.106928] [] sys_select+0x51/0xf0 > Jul 6 06:45:01 coyote kernel: [78748.106931] [] sysenter_do_call+0x12/0x22 > Jul 6 06:45:01 coyote kernel: [78748.106933] Mem-Info: > Jul 6 06:45:01 coyote kernel: [78748.106935] DMA per-cpu: > Jul 6 06:45:01 coyote kernel: [78748.106937] CPU 0: hi: 0, btch: 1 usd: 0 > Jul 6 06:45:01 coyote kernel: [78748.106939] CPU 1: hi: 0, btch: 1 usd: 0 > Jul 6 06:45:01 coyote kernel: [78748.106941] CPU 2: hi: 0, btch: 1 usd: 0 > Jul 6 06:45:01 coyote kernel: [78748.106943] CPU 3: hi: 0, btch: 1 usd: 0 > Jul 6 06:45:01 coyote kernel: [78748.106944] Normal per-cpu: > Jul 6 06:45:01 coyote kernel: [78748.106946] CPU 0: hi: 186, btch: 31 usd: 118 > Jul 6 06:45:01 coyote kernel: [78748.106948] CPU 1: hi: 186, btch: 31 usd: 171 > Jul 6 06:45:01 coyote kernel: [78748.106950] CPU 2: hi: 186, btch: 31 usd: 159 > Jul 6 06:45:01 coyote kernel: [78748.106952] CPU 3: hi: 186, btch: 31 usd: 172 > Jul 6 06:45:01 coyote kernel: [78748.106954] HighMem per-cpu: > Jul 6 06:45:01 coyote kernel: [78748.106955] CPU 0: hi: 186, btch: 31 usd: 56 > Jul 6 06:45:01 coyote kernel: [78748.106957] CPU 1: hi: 186, btch: 31 usd: 20 > Jul 6 06:45:01 coyote kernel: [78748.106959] CPU 2: hi: 186, btch: 31 usd: 53 > Jul 6 06:45:01 coyote kernel: [78748.106961] CPU 3: hi: 186, btch: 31 usd: 180 > Jul 6 06:45:01 coyote kernel: [78748.106965] Active_anon:90702 active_file:136927 inactive_anon:26328 > Jul 6 06:45:01 coyote kernel: [78748.106966] inactive_file:1956 unevictable:25 dirty:4 writeback:0 unstable:0 > Jul 6 06:45:01 coyote kernel: [78748.106967] free:560899 slab:206505 mapped:19048 pagetables:3220 bounce:0 > Jul 6 06:45:01 coyote kernel: [78748.106971] DMA free:3496kB min:64kB low:80kB high:96kB active_anon:0kB > inactive_anon:0kB acti > ve_file:12kB inactive_file:8kB unevictable:0kB present:15804kB pages_scanned:0 all_unreclaimable? yes > Jul 6 06:45:01 coyote kernel: [78748.106983] lowmem_reserve[]: 0 0 25406 25406 > Jul 6 06:45:01 coyote kernel: [78748.106988] HighMem free:2236464kB min:512kB low:3928kB high:7348kB > active_anon:362808kB inact > ive_anon:105308kB active_file:547220kB inactive_file:7704kB unevictable:100kB present:3252052kB pages_scanned:0 > all_unreclaimabl > e? no Normal zone is absent in the above lines. > Jul 6 06:45:01 coyote kernel: [78748.106991] lowmem_reserve[]: 0 0 0 0 > Jul 6 06:45:01 coyote kernel: [78748.106994] DMA: 310*4kB 204*8kB 27*16kB 6*32kB 0*64kB 0*128kB 0*256kB 0*512kB > 0*1024kB 0*2048 > kB 0*4096kB = 3496kB > Jul 6 06:45:01 coyote kernel: [78748.107002] Normal: 1*4kB 0*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 1*512kB > 1*1024kB 1*2048kB > 0*4096kB = 3636kB > Jul 6 06:45:01 coyote kernel: [78748.107009] HighMem: 45888*4kB 44682*8kB 32844*16kB 18479*32kB 6641*64kB 1114*128kB > 37*256kB 1 > *512kB 1*1024kB 0*2048kB 0*4096kB = 2236464kB > Jul 6 06:45:01 coyote kernel: [78748.107017] 139771 total pagecache pages > Jul 6 06:45:01 coyote kernel: [78748.107019] 616 pages in swap cache > Jul 6 06:45:01 coyote kernel: [78748.107021] Swap cache stats: add 7937, delete 7321, find 2362/2515 > Jul 6 06:45:01 coyote kernel: [78748.107023] Free swap = 8360652kB > Jul 6 06:45:01 coyote kernel: [78748.107024] Total swap = 8385912kB > Jul 6 06:45:01 coyote kernel: [78748.121323] 1179632 pages RAM > Jul 6 06:45:01 coyote kernel: [78748.121325] 951810 pages HighMem HighMem zone is 3.7G, which is _too much_ given the total memory is 4G. > Jul 6 06:45:01 coyote kernel: [78748.121327] 146165 pages reserved > Jul 6 06:45:01 coyote kernel: [78748.121328] 149491 pages shared > Jul 6 06:45:01 coyote kernel: [78748.121329] 441615 pages non-shared > Jul 6 06:45:01 coyote kernel: [78748.121332] Out of memory: kill process 2385 (mysqld) score 15451 or a child > Jul 6 06:45:01 coyote kernel: [78748.121334] Killed process 2385 (mysqld) > > and continued to: > > Jul 6 06:45:01 coyote kernel: [78748.137525] Killed process 30192 (spamd) > Jul 6 06:45:01 coyote kernel: [78748.154292] Killed process 2506 (httpd) > Jul 6 06:45:01 coyote kernel: [78748.170851] Killed process 2507 (httpd) > Jul 6 06:45:01 coyote kernel: [78748.187519] Killed process 2508 (httpd) > Jul 6 06:45:01 coyote kernel: [78748.320880] Killed process 2510 (httpd) > Jul 6 06:45:01 coyote kernel: [78748.337529] Killed process 2511 (httpd) > Jul 6 06:45:05 coyote kernel: [78751.467158] Killed process 948 (spamd) > Jul 6 06:45:05 coyote kernel: [78751.483535] Killed process 963 (httpd) > Jul 6 06:45:05 coyote kernel: [78751.500194] Killed process 964 (httpd) > Jul 6 06:45:05 coyote kernel: [78751.502688] Killed process 965 (httpd) > Jul 6 06:45:05 coyote kernel: [78751.506025] Killed process 2512 (httpd) > Jul 6 06:45:05 coyote kernel: [78751.509376] Killed process 2513 (httpd) > Jul 6 06:45:05 coyote kernel: [78751.512713] Killed process 2514 (httpd) > Jul 6 06:45:05 coyote kernel: [78751.516057] Killed process 1995 (krunner_lock) > Jul 6 06:45:05 coyote kernel: [78751.519481] Killed process 3391 (kwin) > Jul 6 06:45:05 coyote kernel: [78751.522674] Killed process 3396 (plasma) > Jul 6 06:45:05 coyote kernel: [78751.526006] Killed process 6931 (kmail) > Jul 6 06:45:05 coyote kernel: [78751.542786] Killed process 16309 (spamd) > Jul 6 06:45:05 coyote kernel: [78751.546042] Killed process 3421 (krunner) > Jul 6 06:45:05 coyote kernel: [78751.549521] Killed process 3357 (klauncher) > Jul 6 06:45:05 coyote kernel: [78751.552760] Killed process 3526 (kcalc) > Jul 6 06:45:05 coyote kernel: [78751.759570] Killed process 3068 (X) > Jul 6 06:45:05 coyote kernel: [78751.765648] Killed process 3389 (ksmserver) Are you running thousands of httpd or other processes? Thanks, Fengguang > The oom started at 6:45:01 this morning. mysqld wasn't doing anything & the > only reason its even started is for mythtv, which is how I found my > pcHDTV-3000 was on the missing list even if occupying a slot. > > I've no idea if the rebuilt (with v4l drivers now) will also crash. > It feels normal. And looks normal in htop's display, using 538M of 4096M, > no swap used yet. > > >From my .config: > > # grep MEM .config > CONFIG_SHMEM=y > # CONFIG_MEMTEST is not set > # CONFIG_NOHIGHMEM is not set > # CONFIG_HIGHMEM4G is not set > CONFIG_HIGHMEM64G=y > CONFIG_HIGHMEM=y > CONFIG_ARCH_FLATMEM_ENABLE=y > CONFIG_ARCH_SPARSEMEM_ENABLE=y > CONFIG_ARCH_SELECT_MEMORY_MODEL=y > CONFIG_SELECT_MEMORY_MODEL=y > CONFIG_FLATMEM_MANUAL=y > # CONFIG_DISCONTIGMEM_MANUAL is not set > # CONFIG_SPARSEMEM_MANUAL is not set > CONFIG_FLATMEM=y > CONFIG_FLAT_NODE_MEM_MAP=y > CONFIG_SPARSEMEM_STATIC=y > CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y > CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y > # CONFIG_BLK_DEV_UMEM is not set > # CONFIG_INPUT_FF_MEMLESS is not set > CONFIG_DEVKMEM=y > CONFIG_FIX_EARLYCON_MEM=y > # CONFIG_HW_RANDOM_TIMERIOMEM is not set > # CONFIG_MEMSTICK is not set > CONFIG_FIRMWARE_MEMMAP=y > CONFIG_DEBUG_MEMORY_INIT=y > CONFIG_HAVE_ARCH_KMEMCHECK=y > CONFIG_STRICT_DEVMEM=y > CONFIG_HAS_IOMEM=y > > In case I have some option miss-set in that, plz advise. > > Thanks. > > -- > Cheers, Gene > "There are four boxes to be used in defense of liberty: > soap, ballot, jury, and ammo. Please use in that order." > -Ed Howdershelt (Author) > The NRA is offering FREE Associate memberships to anyone who wants them. > > > Mal: "Shouldn't have been you - they shoulda hit us." > > Book: "That crossed my mind..." > --"Serenity" > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/