Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752939AbbH1HQG (ORCPT ); Fri, 28 Aug 2015 03:16:06 -0400 Received: from mail-db3on0120.outbound.protection.outlook.com ([157.55.234.120]:41402 "EHLO emea01-db3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752809AbbH1HQD (ORCPT ); Fri, 28 Aug 2015 03:16:03 -0400 X-Greylist: delayed 2031 seconds by postgrey-1.27 at vger.kernel.org; Fri, 28 Aug 2015 03:16:03 EDT From: Steffen Persvold To: Yinghai Lu CC: x86 , LKML Subject: Re: CONFIG_HOLES_IN_ZONE and memory hot plug code on x86_64 Thread-Topic: CONFIG_HOLES_IN_ZONE and memory hot plug code on x86_64 Thread-Index: AQIJZZhDNwHVdxQWiLDsPSNoynCQ4Z2v12aA//+hXoA= Date: Fri, 28 Aug 2015 06:42:07 +0000 Message-ID: <2F8F36EB-D2D8-4DCE-910C-C56FFCEED3BF@numascale.com> References: <26D4DE95-B579-442D-AF7B-469CC4403C51@numascale.com> In-Reply-To: Accept-Language: en-US, nb-NO Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=sp@numascale.com; kernel.org; dkim=none (message not signed) header.d=none;kernel.org; dmarc=none action=none header.from=numascale.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [40.139.113.210] x-microsoft-exchange-diagnostics: 1;DB3PR07MB0524;5:umcuNuxpNxYHgsmoMNEXSMcTyR4gTWo1/te0q3D1XQOKzEDY3jD/KsSnz17t363sTKbr2ZRrrp7r07/A2TjoU4ToX9YSAbYlINJB0TUqi0bP+iZKdO5mJuRzCHAw+m80VKiGljFbbE1ozSVIastwGg==;24:rKnfckDPqMHITHGNvEmJBypXTPEFvqiU1eXWZeWoCfmfhnCzohLFW/hRKESbq1Gan5wOowW6hLoAYRWzzHct23J4w0HiCG8N8bphOk/ZdlA=;20:akeXgXyvIwU9vo9alMXOGSlpqgv214juZ5orJqvH81GNim++cSptqmZ80VVYCEnaRKaaE1V4IgRhFclIgHz3wg== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:DB3PR07MB0524;UriScan:;BCL:0;PCL:0;RULEID:;SRVR:DB3PR07MB0764; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(601004)(8121501046)(5005006)(3002001);SRVR:DB3PR07MB0524;BCL:0;PCL:0;RULEID:;SRVR:DB3PR07MB0524; x-forefront-prvs: 0682FC00E8 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(189002)(377454003)(24454002)(479174004)(199003)(2656002)(83716003)(19580395003)(19580405001)(105586002)(46102003)(64706001)(92566002)(122556002)(101416001)(50986999)(76176999)(54356999)(66066001)(87936001)(33656002)(82746002)(2900100001)(2950100001)(86362001)(40100003)(10400500002)(36756003)(189998001)(5004730100002)(77096005)(106116001)(5007970100001)(102836002)(5002640100001)(62966003)(106356001)(77156002)(68736005)(97736004)(4001540100001)(81156007)(5001960100002)(110136002)(5001860100001)(5001830100001)(104396002)(42262002);DIR:OUT;SFP:1102;SCL:1;SRVR:DB3PR07MB0524;H:DB3PR07MB0524.eurprd07.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: <58933D078EEB924FBC1DE8EDF26EA87D@eurprd07.prod.outlook.com> MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Aug 2015 06:42:07.6529 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4ed3bfc2-5bc0-4223-a1a9-b6c4a7979ada X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR07MB0524 X-Microsoft-Exchange-Diagnostics: 1;DB3PR07MB0764;2:DpwW7YPZpzlWmOmQP3rQJo62NJ5qg9KNt9l5KvaSc9N6Mrn3HM/H6TFGrp1mqVXZMjyVqifWKfLEamKj+F2M6UOV1G4/BKp/qM22okixzY3IX/x7z/ySGeMPXFWp187KR2qXb9tnJ6v1FW4uJlJO22EMOU+kScc+uC/4xuYodT0=;3:KTziPmtWlsP9gFr/fCf4Ibu6qYhSiQpQPbtqLbe/RwOM3nVaJER5i1CySd0oC4+SLisgdz/ScTCx2uCVqYqitHHjp/pgYSnKH9WvKD8iU1gjTi+MejaqZ6ZkLBpODivXuPGj8B7RTAFYC0WBhW2FFA==;25:vB6z7tX4RiG/ZujnRONym3XJOBv1KDI+0fstovOoy+IpgDgU1G8+sp+m8RyJhC/Z//1g2r94FIdY0d9fLOAnDoiJehcbDDN4kf4h43BdI2TIvDbvPA0/huP9b+JI1rfJCVFp+H1lDymXMB0tMTM7zXgjlWzGyZcCYNomq97bPLMJ0B2CLKghHuiseE1R0mgZcLLURx71kOFcLWmi6OyN5HBhF64W08nlVNZ8gW5QelJJ3kjH4Py1Jw1+Vid2/KNrFP//etaEurRlMqVl8fjp4A==;23:Czdx6XAUb06P4ohUNEDzfTC9SXbmLz8+SCKcDhxfwShFz8d85owvVtJ9OvZkNaN788hBZTu4aT6wOlt8btQidV054uSe87fEj5+Afy+dSogvfOx4TIHaiqfBUNRTev36iZU+pk2mlEV1dLYCggxsa6EBjJkpU5Xe8CAW5i43isbK2+CD5xbrS7P3iCwZmpc2 X-OriginatorOrg: numascale.com Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id t7S7GBWx005065 Content-Length: 2874 Lines: 50 On 27/08/15 22:20 , "yhlu.kernel@gmail.com on behalf of Yinghai Lu" wrote: >On Fri, Jun 26, 2015 at 4:31 PM, Steffen Persvold wrote: >> We’ve encountered an issue in a special case where we have a sparse E820 map [1]. >> >> Basically the memory hotplug code is causing a “kernel paging request” BUG [2]. > >the trace does not look like hotplug path. > >> >> By instrumenting the function register_mem_sect_under_node() in drivers/base/node.c we see that it is called two times with the same struct memory_block argument : >> >> [ 1.901463] register_mem_sect_under_node: start = 80, end = 8f, nid = 0 >> [ 1.908129] register_mem_sect_under_node: start = 80, end = 8f, nid = 1 > >Can you post whole log with SRAT related info? I can probably reproduce again and get full logs when I get run time on the system again, but here’s some output that we saved in our internal Jira case : [ 0.000000] NUMA: Initialized distance table, cnt=6 [ 0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0xd7ffffff] -> [mem 0x00000000-0xd7ffffff] [ 0.000000] NUMA: Node 0 [mem 0x00000000-0xd7ffffff] + [mem 0x100000000-0x427ffffff] -> [mem 0x00000000-0x427ffffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x407fe3000-0x407ffffff] [ 0.000000] NODE_DATA(1) allocated [mem 0x807fe3000-0x807ffffff] [ 0.000000] NODE_DATA(2) allocated [mem 0xc07fe3000-0xc07ffffff] [ 0.000000] NODE_DATA(3) allocated [mem 0x1007fe3000-0x1007ffffff] [ 0.000000] NODE_DATA(4) allocated [mem 0x1407fe3000-0x1407ffffff] [ 0.000000] NODE_DATA(5) allocated [mem 0x1807fdd000-0x1807ff9fff] [ 0.000000] [ffffea0000000000-ffffea00101fffff] PMD -> [ffff8803f8600000-ffff880407dfffff] on node 0 [ 0.000000] [ffffea0010a00000-ffffea00201fffff] PMD -> [ffff8807f8600000-ffff880807dfffff] on node 1 [ 0.000000] [ffffea0020a00000-ffffea00301fffff] PMD -> [ffff880bf8600000-ffff880c07dfffff] on node 2 [ 0.000000] [ffffea0030a00000-ffffea00401fffff] PMD -> [ffff880ff8600000-ffff881007dfffff] on node 3 [ 0.000000] [ffffea0040a00000-ffffea00501fffff] PMD -> [ffff8813f8600000-ffff881407dfffff] on node 4 [ 0.000000] [ffffea0050a00000-ffffea00601fffff] PMD -> [ffff8817f7e00000-ffff8818075fffff] on node 5 If I remember correctly there was a mix of 4GB and 8GB DIMMs populated on this system. In addition the firmware reserved 512MByte at the end of each memory controllers physical range (hence the reserved ranges in the e820 map). Note: this was with 4.1.0 vanilla so it could be obsolete now with 4.2-rc. I have not yet tested with your latest patches that you and Tony discussed. Cheers, Steffen ????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay?ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m???? ????????I?