Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1752041AbbEDVaz (ORCPT <rfc822;w@1wt.eu>);
	Mon, 4 May 2015 17:30:55 -0400
Received: from mail.linuxfoundation.org ([140.211.169.12]:41293 "EHLO
	mail.linuxfoundation.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1750992AbbEDVas (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Mon, 4 May 2015 17:30:48 -0400
Date: Mon, 4 May 2015 14:30:46 -0700
From: Andrew Morton <akpm@linux-foundation.org>
To: Waiman Long <waiman.long@hp.com>
Cc: Mel Gorman <mgorman@suse.de>, Nathan Zimmer <nzimmer@sgi.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Scott Norton <scott.norton@hp.com>,
        Daniel J Blueman <daniel@numascale.com>, Linux-MM <linux-mm@kvack.org>,
        LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 0/13] Parallel struct page initialisation v4
Message-Id: <20150504143046.9404c572486caf71bdef0676@linux-foundation.org>
In-Reply-To: <554415B1.2050702@hp.com>
References: <1430231830-7702-1-git-send-email-mgorman@suse.de>
	<554030D1.8080509@hp.com>
	<5543F802.9090504@hp.com>
	<554415B1.2050702@hp.com>
X-Mailer: Sylpheed 3.4.1 (GTK+ 2.24.23; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 3097
Lines: 66

On Fri, 01 May 2015 20:09:21 -0400 Waiman Long <waiman.long@hp.com> wrote:

> On 05/01/2015 06:02 PM, Waiman Long wrote:
> >
> > Bad news!
> >
> > I tried your patch on a 24-TB DragonHawk and got an out of memory 
> > panic. The kernel log messages were:
>
> ...
>
> > [   81.360287]  [<ffffffff8151b0c9>] dump_stack+0x68/0x77
> > [   81.365942]  [<ffffffff8151ae1e>] panic+0xb9/0x219
> > [   81.371213]  [<ffffffff810785c3>] ? 
> > __blocking_notifier_call_chain+0x63/0x80
> > [   81.378971]  [<ffffffff811384ce>] __out_of_memory+0x34e/0x350
> > [   81.385292]  [<ffffffff811385ee>] out_of_memory+0x5e/0x90
> > [   81.391230]  [<ffffffff8113ce9e>] __alloc_pages_slowpath+0x6be/0x740
> > [   81.398219]  [<ffffffff8113d15c>] __alloc_pages_nodemask+0x23c/0x250
> > [   81.405212]  [<ffffffff81186346>] kmem_getpages+0x56/0x110
> > [   81.411246]  [<ffffffff81187f44>] fallback_alloc+0x164/0x200
> > [   81.417474]  [<ffffffff81187cfd>] ____cache_alloc_node+0x8d/0x170
> > [   81.424179]  [<ffffffff811887bb>] kmem_cache_alloc_trace+0x17b/0x240
> > [   81.431169]  [<ffffffff813d5f3a>] init_memory_block+0x3a/0x110
> > [   81.437586]  [<ffffffff81b5f687>] memory_dev_init+0xd7/0x13d
> > [   81.443810]  [<ffffffff81b5f2af>] driver_init+0x2f/0x37
> > [   81.449556]  [<ffffffff81b1599b>] do_basic_setup+0x29/0xd5
> > [   81.455597]  [<ffffffff81b372c4>] ? sched_init_smp+0x140/0x147
> > [   81.462015]  [<ffffffff81b15c55>] kernel_init_freeable+0x20e/0x297
> > [   81.468815]  [<ffffffff81512ea0>] ? rest_init+0x80/0x80
> > [   81.474565]  [<ffffffff81512ea9>] kernel_init+0x9/0xf0
> > [   81.480216]  [<ffffffff8151f788>] ret_from_fork+0x58/0x90
> > [   81.486156]  [<ffffffff81512ea0>] ? rest_init+0x80/0x80
> > [   81.492350] ---[ end Kernel panic - not syncing: Out of memory and 
> > no killable processes...
> > [   81.492350]
> >
> > -Longman
> 
> I increased the pre-initialized memory per node in update_defer_init() 
> of mm/page_alloc.c from 2G to 4G. Now I am able to boot the 24-TB 
> machine without error. The 12-TB has 0.75TB/node, while the 24-TB 
> machine has 1.5TB/node. I would suggest something like pre-initializing 
> 1G per 0.25TB/node. In this way, it will scale properly with the memory 
> size.

We're using more than 2G before we've even completed do_basic_setup()? 
Where did it all go?

> Before the patch, the boot time from elilo prompt to ssh login was 694s. 
> After the patch, the boot up time was 346s, a saving of 348s (about 50%).

Having to guesstimate the amount of memory which is needed for a
successful boot will be painful.  Any number we choose will be wrong
99% of the time.

If the kswapd threads have started, all we need to do is to wait: take
a little nap in the allocator's page==NULL slowpath.

I'm not seeing any reason why we can't start kswapd much earlier -
right at the start of do_basic_setup()?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/