Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <9436d033-579b-55fa-9b00-6f4b661c2dd7@linux.microsoft.com>
Date:   Wed, 9 Feb 2022 13:47:51 -0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.1
Subject: Re: [arm64] kernel boot slowdown in v5.10.19 -> v5.10.42 update
Content-Language: en-US
From:   Vijay Balakrishna <vijayb@linux.microsoft.com>
To:     Catalin Marinas <catalin.marinas@arm.com>
Cc:     Linux kernel mailing list <linux-kernel@vger.kernel.org>,
        linux-arm-kernel@lists.infradead.org,
        Nicolas Saenz Julienne <nsaenz@kernel.org>
References: <9a34ee9b-0ede-30a6-0898-d32fe81d5b0c@linux.microsoft.com>
 <YfPcNy3JcnwuJNMx@arm.com>
 <7e43a7e9-f299-1d6d-8b61-425793abcc3e@linux.microsoft.com>
In-Reply-To: <7e43a7e9-f299-1d6d-8b61-425793abcc3e@linux.microsoft.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk


On 1/28/2022 10:03 AM, Vijay Balakrishna wrote:
> 
> 
> On 1/28/2022 4:06 AM, Catalin Marinas wrote:
>> On Mon, Jan 24, 2022 at 03:03:48PM -0800, Vijay Balakrishna wrote:
>>> We noticed 150ms kernel boot slowdown back in June, 2021, when moving 
>>> from
>>> v5.10.19 to v5.10.42.  This on a 8GB SoC.  Only recently we investigated
>>> this issue and found the regression is introduced by a change in 
>>> map_mem()
>>> (paging_init() -> map_mem() -> __map_memblock(), in particular "map 
>>> all the
>>> memory banks" for loop) by patch
>>>
>>> 2687275a5843d1089687f08fc64eb3f3b026a169
>>> arm64: Force NO_BLOCK_MAPPINGS if crashkernel reservation is required
>>>
>>> above is a follow up to
>>>
>>> 0a30c53573b07d5561457e41fb0ab046cd857da5
>>> arm64: mm: Move reserve_crashkernel() into mem_init())
>>>
>>> which deferred crashkernel reservation into mem_init().
>>>
>>> The ~150ms slowdown disappears on booting without "crashkernel=.." on 
>>> kernel
>>> command-line.
>>
>> Is CONFIG_RODATA_FULL_DEFAULT_ENABLED set in your config file? It may
>> cause the same slowdown. I suspect it's because we end up mapping the
>> RAM at the page granularity so more loops and slightly higher TLB
>> pressure. Not sure we can do much about >
> 
> Catalin,
> 
> Thanks for your reply.
> 
> CONFIG_RODATA_FULL_DEFAULT_ENABLED is not set in our config for boot 
> time performance reasons.

 From following patches, the main source of deferring crash kernel 
reservation is for Raspberry Pi 4, which has no IOMMU.

1a8e1cef7603e218339ac63cb3178b25554524e5
arm64: use both ZONE_DMA and ZONE_DMA32

8424ecdde7df99d5426e1a1fd9f0fb36f4183032
arm64: mm: Set ZONE_DMA size based on devicetree's dma-ranges

0a30c53573b07d5561457e41fb0ab046cd857da5
arm64: mm: Move reserve_crashkernel() into mem_init())

2687275a5843d1089687f08fc64eb3f3b026a169
arm64: Force NO_BLOCK_MAPPINGS if crashkernel reservation is required

However above changes introduced slow boot (regression) for platforms 
with IOMMU, for example as I mentioned earlier ~150ms for us.  We don't 
have following configs set for our SoC --

# CONFIG_ZONE_DMA is not set
# CONFIG_ZONE_DMA32 is not set

Can we restore crash kernel reservation (no deferring to mem_init()) for 
platforms with no DMA memory zones?

I can submit a patch.

Thanks,
Vijay