Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755191Ab3HEIAh (ORCPT ); Mon, 5 Aug 2013 04:00:37 -0400 Received: from fgwmail5.fujitsu.co.jp ([192.51.44.35]:45739 "EHLO fgwmail5.fujitsu.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755017Ab3HEIAf (ORCPT ); Mon, 5 Aug 2013 04:00:35 -0400 X-SecurityPolicyCheck: OK by SHieldMailChecker v2.0.1 X-SHieldMailCheckerPolicyVersion: FJ-ISEC-20120718-3 Message-ID: <51FF5B58.7050701@jp.fujitsu.com> Date: Mon, 5 Aug 2013 16:59:20 +0900 From: Yasuaki Ishimatsu User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: Toshi Kani , "Rafael J. Wysocki" CC: , , , , , Subject: Re: Cannot hot remove a memory device References: <51FA1E41.20304@jp.fujitsu.com> <2124016.41i4c3lbD4@vostro.rjw.lan> <1375488280.10300.124.camel@misato.fc.hp.com> <3856529.sQXzdEem7D@vostro.rjw.lan> <1375576646.10300.150.camel@misato.fc.hp.com> <51FF2368.9060206@jp.fujitsu.com> In-Reply-To: <51FF2368.9060206@jp.fujitsu.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-SecurityPolicyCheck-GC: OK by FENCE-Mail Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 6843 Lines: 153 (2013/08/05 13:00), Yasuaki Ishimatsu wrote: > (2013/08/04 9:37), Toshi Kani wrote: >> On Sat, 2013-08-03 at 03:01 +0200, Rafael J. Wysocki wrote: >>> On Friday, August 02, 2013 06:04:40 PM Toshi Kani wrote: >>>> On Sat, 2013-08-03 at 01:43 +0200, Rafael J. Wysocki wrote: >>>>> On Friday, August 02, 2013 03:46:15 PM Toshi Kani wrote: >>>>>> On Thu, 2013-08-01 at 23:43 +0200, Rafael J. Wysocki wrote: >>>>>>> Hi, >>>>>>> >>>>>>> Thanks for your report. >>>>>>> >>>>>>> On Thursday, August 01, 2013 05:37:21 PM Yasuaki Ishimatsu wrote: >>>>>>>> By following commit, I cannot hot remove a memory device. >>>>>>>> >>>>>>>> ACPI / memhotplug: Bind removable memory blocks to ACPI device nodes >>>>>>>> commit e2ff39400d81233374e780b133496a2296643d7d >>>>>>>> >>>>>>>> Details are follows: >>>>>>>> When I add a memory device, acpi_memory_enable_device() always fails >>>>>>>> as follows: >>>>>>>> >>>>>>>> ... >>>>>>>> [ 1271.114116] [ffffea121c400000-ffffea121c7fffff] PMD -> [ffff880813c00000-ffff880813ffffff] on node 3 >>>>>>>> [ 1271.128682] [ffffea121c800000-ffffea121cbfffff] PMD -> [ffff880813800000-ffff880813bfffff] on node 3 >>>>>>>> [ 1271.143298] [ffffea121cc00000-ffffea121cffffff] PMD -> [ffff880813000000-ffff8808133fffff] on node 3 >>>>>>>> [ 1271.157799] [ffffea121d000000-ffffea121d3fffff] PMD -> [ffff880812c00000-ffff880812ffffff] on node 3 >>>>>>>> [ 1271.172341] [ffffea121d400000-ffffea121d7fffff] PMD -> [ffff880812800000-ffff880812bfffff] on node 3 >>>>>>>> [ 1271.186872] [ffffea121d800000-ffffea121dbfffff] PMD -> [ffff880812400000-ffff8808127fffff] on node 3 >>>>>>>> [ 1271.201481] [ffffea121dc00000-ffffea121dffffff] PMD -> [ffff880812000000-ffff8808123fffff] on node 3 >>>>>>>> [ 1271.216041] [ffffea121e000000-ffffea121e3fffff] PMD -> [ffff880811c00000-ffff880811ffffff] on node 3 >>>>>>>> [ 1271.230623] [ffffea121e400000-ffffea121e7fffff] PMD -> [ffff880811800000-ffff880811bfffff] on node 3 >>>>>>>> [ 1271.245148] [ffffea121e800000-ffffea121ebfffff] PMD -> [ffff880811400000-ffff8808117fffff] on node 3 >>>>>>>> [ 1271.259683] [ffffea121ec00000-ffffea121effffff] PMD -> [ffff880811000000-ffff8808113fffff] on node 3 >>>>>>>> [ 1271.274194] [ffffea121f000000-ffffea121f3fffff] PMD -> [ffff880810c00000-ffff880810ffffff] on node 3 >>>>>>>> [ 1271.288764] [ffffea121f400000-ffffea121f7fffff] PMD -> [ffff880810800000-ffff880810bfffff] on node 3 >>>>>> >>>>>> It appears that each memory object only has 64MB of memory. This is >>>>>> less than the memory block size, which is 128MB. This means that a >>>>>> single memory block associates with two ACPI memory device objects. >>>>> >>>>> That'd be bad. >>>>> >>>>> How did that work before if that indeed is the case? >>>> >>>> Well, it looks to me that it has never worked before... >>>> >>>>>>>> ... >>>>>>>> [ 1271.325841] acpi PNP0C80:03: acpi_memory_enable_device() error >>>>>>> >>>>>>> Well, the only new way acpi_memory_enable_device() can fail after that commit >>>>>>> is a failure in acpi_bind_memory_blocks(). >>>>>> >>>>>> Agreed. >>>>>> >>>>>>> This means that either handle is NULL, which I think we can exclude, because >>>>>>> acpi_memory_enable_device() wouldn't be called at all if that were the case, or >>>>>>> there's a more subtle error in acpi_bind_one(). >>>>>>> >>>>>>> One that comes to mind is that we may be calling acpi_bind_one() twice for the >>>>>>> same memory region, in which it will trigger -EINVAL from the sanity check in >>>>>>> there. >>>>>> >>>>>> I think it fails with -EINVAL at the place with dev_warn(dev, "ACPI >>>>>> handle is already set\n"). When two ACPI memory objects associate with >>>>>> a same memory block, the bind procedure of the 2nd ACPI memory object >>>>>> sees that ACPI_HANDLE(dev) is already set to the 1st ACPI memory object. >>>>> >>>>> That sound's plausible, but I wonder how we can fix that? >>>>> >>>>> There's no way for a single physical device to have two different ACPI >>>>> "companions". It looks like the memory blocks should be 64 M each in that >>>>> case. Or we need to create two child devices for each memory block and >>>>> associate each of them with an ACPI object. That would lead to complications >>>>> in the user space interface, though. >>>> >>>> Right. Even bigger issue is that I do not think __add_pages() and >>>> __remove_pages() can add / delete a memory chunk that is less than >>>> 128MB. 128MB is the granularity of them. So, we may just have to fail >>>> this case gracefully. >>> >>> Sigh. >>> >>> BTW, why do you think they are 64 M each (it's late and I'm obviously tired)? >> >> Oops! Sorry, I had confused the above messages with the one in >> init_memory_mapping(), which shows a memory range being added, i.e. the >> size of an ACPI memory device object. But the above messages actually >> came from vmemmap_populate_hugepages(), which was called during boot-up. >> So, these messages have nothing to do with ACPI memory device objects. >> And even worse, I do not seem to be able to count a number of zeros... >> In the above messages, each memory range is 4MB (0x400000), not 64MB >> (0x4000000)... My bad. :-( >> >> So, while we may still need to do something for the less-than-128MB >> issue, Yasuaki may be hitting a different one. Let's wait for Yasuaki >> to give us more info. > > acpi_bind_memory_blocks() failed with -ENOSPC. > > int acpi_bind_one(struct device *dev, acpi_handle handle) > { > ... > /* allocate physical node id according to physical_node_id_bitmap */ > physical_node->node_id = > find_first_zero_bit(acpi_dev->physical_node_id_bitmap, > ACPI_MAX_PHYSICAL_NODE); > if (physical_node->node_id >= ACPI_MAX_PHYSICAL_NODE) { > retval = -ENOSPC; => here > goto err_free; > } > > When adding memory device, acpi_bind_memroy_blocks() calls acpi_bind_one() > "memory device size / 128MiB" times. So ACPI_MAX_PHYSICAL_NODE need to > be set "memory device size / 128MiB" or more. But ACPI_MAX_PHYSICAL_NODE is 32. > So acpi_bind_memory_blocks() always failed with -ENOSPC. > > I'll test again after increasing ACPI_MAX_PHYSICAL_NODE to enough size. Additional info: When I increased ACPI_MAX_PHYSICAL_NODE to 1024 and change size of acpi_device_physical_node->node_it into u32, I could hot remove memory device. But even if ACPI_MAX_PHYSICAL_NODE is set to 1024, same problem will occurs since it just supports 124GiB memory. So we need a way to change ACPI_MAX_PHYSICAL_NODE dynamically. Thanks, Yasuaki Ishimatsu > > Thanks, > Yasuaki Ishimatsu > >> >> Thanks, >> -Toshi >> >> >> >> >> >> >> >> >> > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/