Message-ID: <1371655111.22206.29.camel@misato.fc.hp.com>
Subject: Re: [Part3 PATCH v2 0/4] Support hot-remove local pagetable pages.
From: Toshi Kani <toshi.kani@hp.com>
To: Tang Chen <tangchen@cn.fujitsu.com>
Cc: Vasilis Liaskovitis <vasilis.liaskovitis@profitbricks.com>,
        yinghai@kernel.org, tglx@linutronix.de, mingo@elte.hu, hpa@zytor.com,
        akpm@linux-foundation.org, tj@kernel.org, trenn@suse.de,
        jiang.liu@huawei.com, wency@cn.fujitsu.com, laijs@cn.fujitsu.com,
        isimatu.yasuaki@jp.fujitsu.com, mgorman@suse.de, minchan@kernel.org,
        mina86@mina86.com, gong.chen@linux.intel.com, lwoodman@redhat.com,
        riel@redhat.com, jweiner@redhat.com, prarit@redhat.com, x86@kernel.org,
        linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-mm@kvack.org
Date: Wed, 19 Jun 2013 09:18:31 -0600
In-Reply-To: <51C15DC2.3030501@cn.fujitsu.com>
References: <1371128636-9027-1-git-send-email-tangchen@cn.fujitsu.com>
	 <20130618170515.GC4553@dhcp-192-168-178-175.profitbricks.localdomain>
	 <51C15DC2.3030501@cn.fujitsu.com>
Content-Type: text/plain; charset="UTF-8"
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 3231
Lines: 74

On Wed, 2013-06-19 at 15:29 +0800, Tang Chen wrote:
> Hi Vasilis, Yinghai,
> 
> On 06/19/2013 01:05 AM, Vasilis Liaskovitis wrote:
> ......
> >
> > This could be a design problem of part3: if we allow local pagetable memory
> > to not be offlined but allow the offlining to return successfully, then
> > hot-remove is going to succeed. But the direct mapped pagetable pages are still
> > mapped in the kernel. The hot-removed memblocks will suddenly disappear (think
> > physical DIMMs getting disabled in real hardware, or in a VM case the
> > corresponding guest memory getting freed from the emulator e.g. qemu/kvm). The
> > system can crash as a result.
> >
> 
> Yes. Since the pagetable pages is only allocated to local node, a node may
> have more than one device, hot-remove only one memory device could be
> problematic.
> 
> But I think it will work if we hot-remove a whole node. I should have
> mentioned it. And sorry for the not fully test.
> 
> I think allocating pagetable pages to local device will resolve this problem.
> And need to restructure this patch-set.
> 
> > I think these local pagetables do need to be unmapped from kernel, offlined and
> > removed somehow - otherwise hot-remove should fail. Could they be migrated
> > alternatively e.g. to node 0 memory?  But Iiuc direct mapped pages cannot be
> > migrated, correct?
> 
> I think we have unmapped the local pagetables. in functions
> free_pud/pmd/pte_table(), we cleared pud, pmd, and pte. We just didn't
> free the pagetable pages to buddy.
> 
> But when we are not hot-removing the whole node, it is still problematic.
> This is true, and it is my design problem.
> 
> >
> > What is the original reason for local node pagetable allocation with regards
> > to memory hotplug? I assume we want to have hotplugged nodes use only their local
> > memory, so that there are no inter-node memory dependencies for hot-add/remove.
> > Are there other reasons that I am missing?
> 
> I think the original reason to do local node pagetable is to improve 
> performance.
> Using local pagetable, vmemmap and so on will be faster.
> 
> But actually I think there is no particular reason to implement memory hot-remove
> and local node pagetable at the same time. And before this patch-set, I also
> suggested once that implement memory hot-remove first, and then improve it to
> local pagetable. But Yinghai has done the local pagetable work in has 
> patches (part1).
> And my work is based on his patches. So I just did it.
> 
> But obviously it is more complicated than I thought.
> 
> And now, it seems tj has some more thinking on part1.
> 
> So how about the following plan:
> 1. Implement arranging hotpluggable memory with SRAT first, without 
> local pagetable.
>     (The main work in part2. And of course, need some patches in part1.)
> 2. Do the local device pagetable work, not local node.
> 3. Improve memory hotplug to support local device pagetable.

Sounds like a good plan.  Thanks Tang!
-Toshi


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/