Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
To:     Mike Rapoport <rppt@linux.ibm.com>
Cc:     "Liang, Liang (Leo)" <Liang.Liang@amd.com>,
        "Deucher, Alexander" <Alexander.Deucher@amd.com>,
        linux-kernel@vger.kernel.org,
        amd-gfx list <amd-gfx@lists.freedesktop.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        "Huang, Ray" <Ray.Huang@amd.com>,
        "Koenig, Christian" <Christian.Koenig@amd.com>,
        "Rafael J. Wysocki" <rafael@kernel.org>,
        George Kennedy <george.kennedy@oracle.com>
References: <MW3PR12MB4537B49678884A1EB1F75AB5F36E9@MW3PR12MB4537.namprd12.prod.outlook.com>
 <0AE49D98-171A-42B9-9CFC-9193A9BD3346@redhat.com>
 <YEzCm/Uwvw7kKpd7@linux.ibm.com>
From:   David Hildenbrand <david@redhat.com>
Organization: Red Hat GmbH
Subject: Re: slow boot with 7fef431be9c9 ("mm/page_alloc: place pages to tail
 in __free_pages_core()")
Message-ID: <22437770-956e-f7b4-a8f6-3f1cc28c3ec2@redhat.com>
Date:   Mon, 15 Mar 2021 14:04:09 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.8.0
MIME-Version: 1.0
In-Reply-To: <YEzCm/Uwvw7kKpd7@linux.ibm.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 13.03.21 14:48, Mike Rapoport wrote:
> Hi,
> 
> On Sat, Mar 13, 2021 at 10:05:23AM +0100, David Hildenbrand wrote:
>>> Am 13.03.2021 um 05:04 schrieb Liang, Liang (Leo) <Liang.Liang@amd.com>:
>>>
>>> Hi David,
>>>
>>> Which benchmark tool you prefer? Memtest86+ or else?
>>
>> Hi Leo,
>>
>> I think you want something that runs under Linux natively.
>>
>> I‘m planning on coding up a kernel module to walk all 4MB pages in the
>> freelists and perform a stream benchmark individually. Then we might be
>> able to identify the problematic range - if there is a problematic range :)
> 
> My wild guess would be that the pages that are now at the head of free
> lists have wrong caching enabled. Might be worth checking in your test
> module.

I hacked something up real quick:

https://github.com/davidhildenbrand/kstream

Only briefly tested inside a VM. The output looks something like

[...]
[ 8396.432225] [0x0000000045800000 - 0x0000000045bfffff] 25322 MB/s / 
38948 MB/s
[ 8396.448749] [0x0000000045c00000 - 0x0000000045ffffff] 24481 MB/s / 
38946 MB/s
[ 8396.465197] [0x0000000046000000 - 0x00000000463fffff] 24892 MB/s / 
39170 MB/s
[ 8396.481552] [0x0000000046400000 - 0x00000000467fffff] 25222 MB/s / 
39156 MB/s
[ 8396.498012] [0x0000000046800000 - 0x0000000046bfffff] 24416 MB/s / 
39159 MB/s
[ 8396.514397] [0x0000000046c00000 - 0x0000000046ffffff] 25469 MB/s / 
38940 MB/s
[ 8396.530849] [0x0000000047000000 - 0x00000000473fffff] 24885 MB/s / 
38734 MB/s
[ 8396.547195] [0x0000000047400000 - 0x00000000477fffff] 25458 MB/s / 
38941 MB/s
[...]

The benchmark allocates one 4 MiB chunk at a time and runs a simplified 
STREAM benchmark a) without flushing caches b) flushing caches before 
every memory access.

It would be great if you could run that with the *old behavior* kernel 
(IOW, without 7fef431be9c9), so we might still be lucky to catch the 
problematic area in the freelist.

Let's see if that will indicate anything.

-- 
Thanks,

David / dhildenb