Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Rahul Gopakumar <gopakumarr@vmware.com>
To:     "bhe@redhat.com" <bhe@redhat.com>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
CC:     "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
        "natechancellor@gmail.com" <natechancellor@gmail.com>,
        "ndesaulniers@google.com" <ndesaulniers@google.com>,
        "clang-built-linux@googlegroups.com" 
        <clang-built-linux@googlegroups.com>,
        "rostedt@goodmis.org" <rostedt@goodmis.org>,
        Rajender M <manir@vmware.com>,
        Yiu Cho Lau <lauyiuch@vmware.com>,
        Peter Jonasson <pjonasson@vmware.com>,
        Venkatesh Rajaram <rajaramv@vmware.com>
Subject: Performance regressions in "boot_time" tests in Linux 5.8 Kernel
Thread-Topic: Performance regressions in "boot_time" tests in Linux 5.8 Kernel
Thread-Index: AQHWnjk5ihPzOwsVE02I7/jG5KTBzw==
Date:   Fri, 9 Oct 2020 13:15:42 +0000
Message-ID: <DM6PR05MB52921FF90FA01CC337DD23A1A4080@DM6PR05MB5292.namprd05.prod.outlook.com>
Accept-Language: en-GB, en-US
Content-Language: en-GB
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 96a3a9e8-bebd-4920-b50b-08d86c556d0b
X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Oct 2020 13:15:42.8619
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: gnTnbvA9rAsXh01hKh6EnqW+eMOdkr0bTdDy4Ox+0qyJx9R3dCYqtvlpiYk86xSPgQUPKNJfLGveH4G4GkuGMA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR05MB6441
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

As part of VMware's performance regression testing for Linux Kernel=0A=
upstream releases,=A0we identified boot time increase when comparing=0A=
Linux 5.8 kernel against Linux 5.7 kernel.=A0Increase in boot time is=0A=
noticeable on VM with a **large amount of memory**.=0A=
=A0=0A=
In our test cases, it's noticeable with memory 1TB and more, whereas=0A=
there was no major=A0difference noticed in testcases with <1TB.=0A=
=A0=0A=
On bisecting between 5.7 and 5.8, we found the following commit from=A0=0A=
=93Baoquan He=94 to be=A0the cause of boot time increase in big VM test cas=
es.=0A=
=A0=0A=
-------------------------------------=0A=
=A0=0A=
commit 73a6e474cb376921a311786652782155eac2fdf0=0A=
Author: Baoquan He <bhe@redhat.com>=0A=
Date: Wed Jun 3 15:57:55 2020 -0700=0A=
=A0=0A=
mm: memmap_init: iterate over memblock regions rather that check each PFN=
=0A=
=A0=0A=
When called during boot the memmap_init_zone() function checks if each PFN=
=0A=
is valid and actually belongs to the node being initialized using=0A=
early_pfn_valid() and early_pfn_in_nid().=0A=
=A0=0A=
Each such check may cost up to O(log(n)) where n is the number of memory=0A=
banks, so for large amount of memory overall time spent in early_pfn*()=0A=
becomes substantial.=0A=
=A0=0A=
-------------------------------------=0A=
=A0=0A=
For boot time test, we used RHEL 8.1 as the guest OS.=0A=
VM config is 84 vcpu and 1TB vRAM.=0A=
=A0=0A=
Here are the actual performance numbers.=0A=
=A0=0A=
5.7 GA - 18.17 secs=0A=
Baoquan's commit - 21.6 secs (-16% increase in time)=0A=
=A0=0A=
From dmesg logs, we can see significant time delay around memmap.=0A=
=A0=0A=
Refer below logs.=0A=
=A0=0A=
Good commit=0A=
=A0=0A=
[0.033176] Normal zone: 1445888 pages used for memmap=0A=
[0.033176] Normal zone: 89391104 pages, LIFO batch:63=0A=
[0.035851] ACPI: PM-Timer IO Port: 0x448=0A=
=A0=0A=
Problem commit=0A=
=A0=0A=
[0.026874] Normal zone: 1445888 pages used for memmap=0A=
[0.026875] Normal zone: 89391104 pages, LIFO batch:63=0A=
[2.028450] ACPI: PM-Timer IO Port: 0x448=0A=
=A0=0A=
We did some analysis, and it looks like with the problem commit it's=0A=
not deferring the memory initialization to a later stage and it's =0A=
initializing the huge chunk of memory in serial - during the boot-up=0A=
time. =A0Whereas with the good commit, it was able to defer the=0A=
initialization of the memory when it could be done in parallel.=0A=
=0A=
=0A=
Rahul Gopakumar=0A=
Performance=A0Engineering=0A=
VMware, Inc.=