Received-SPF: pass (google.com: domain of linux-kernel+bounces-7238-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <65b27029-28be-4744-85c3-6d3aa935aa03@arm.com>
Date: Wed, 20 Dec 2023 15:59:26 +0000
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v4 02/16] mm: Batch-copy PTE ranges during fork()
Content-Language: en-GB
To: David Hildenbrand <david@redhat.com>,
 Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>,
 Ard Biesheuvel <ardb@kernel.org>, Marc Zyngier <maz@kernel.org>,
 Oliver Upton <oliver.upton@linux.dev>, James Morse <james.morse@arm.com>,
 Suzuki K Poulose <suzuki.poulose@arm.com>, Zenghui Yu
 <yuzenghui@huawei.com>, Andrey Ryabinin <ryabinin.a.a@gmail.com>,
 Alexander Potapenko <glider@google.com>,
 Andrey Konovalov <andreyknvl@gmail.com>, Dmitry Vyukov <dvyukov@google.com>,
 Vincenzo Frascino <vincenzo.frascino@arm.com>,
 Andrew Morton <akpm@linux-foundation.org>,
 Anshuman Khandual <anshuman.khandual@arm.com>,
 Matthew Wilcox <willy@infradead.org>, Yu Zhao <yuzhao@google.com>,
 Mark Rutland <mark.rutland@arm.com>, Kefeng Wang
 <wangkefeng.wang@huawei.com>, John Hubbard <jhubbard@nvidia.com>,
 Zi Yan <ziy@nvidia.com>, Barry Song <21cnbao@gmail.com>,
 Alistair Popple <apopple@nvidia.com>, Yang Shi <shy828301@gmail.com>
Cc: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org,
 linux-kernel@vger.kernel.org
References: <20231218105100.172635-1-ryan.roberts@arm.com>
 <7c0236ad-01f3-437f-8b04-125d69e90dc0@redhat.com>
 <9a58b1a2-2c13-4fa0-8ffa-2b3d9655f1b6@arm.com>
 <28968568-f920-47ac-b6fd-87528ffd8f77@redhat.com>
 <10b0b562-c1c0-4a66-9aeb-a6bff5c218f6@arm.com>
 <8f8023cb-3c31-4ead-a9e6-03a10e9490c6@redhat.com>
 <da16a7e5-76dd-4150-9ade-54b0d227a1e1@arm.com>
 <699cb1db-51eb-460e-9ceb-1ce08ca03050@redhat.com>
 <da29a4c6-61f6-4203-9c82-9ce6e1c32552@arm.com>
 <2a8c5b6c-f5ae-43b2-99aa-6d10e79b76e1@redhat.com>
 <ade26f27-03af-4ad7-ad81-38b482f7572c@arm.com>
 <3194b8a5-3f72-4d9e-a267-fbdad32ad864@redhat.com>
 <f2f420cf-678d-466d-ac30-bc8251f16632@arm.com>
 <9f99a3ca-051e-4b1b-81e9-8456d8e422ad@redhat.com>
 <c1913fa2-d313-4f05-a649-da69b6d31d2a@arm.com>
 <c244707c-8c0c-4593-ab45-d8f165dbc5ee@redhat.com>
 <ba6a5c11-34fb-40ca-b24a-f51a207464f2@arm.com>
 <5fcbf405-7e62-4b38-acc4-a9dd8cc91214@redhat.com>
 <396cae5d-70e4-449f-af6c-2348b720d3a3@arm.com>
 <cfd88172-1624-4966-92c6-5197b26a820b@redhat.com>
From: Ryan Roberts <ryan.roberts@arm.com>
In-Reply-To: <cfd88172-1624-4966-92c6-5197b26a820b@redhat.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

On 20/12/2023 15:35, David Hildenbrand wrote:
> On 20.12.23 16:05, Ryan Roberts wrote:
>> On 20/12/2023 14:00, David Hildenbrand wrote:
>>> [...]
>>>
>>>>>>
>>>>>
>>>>> gcc version 13.2.1 20231011 (Red Hat 13.2.1-4) (GCC)
>>>>>
>>>>>   From Fedora 38. So "a bit" newer :P
>>>>>
>>>>
>>>> I'll retry with newer toolchain.
>>>>
>>>> FWIW, with the code fix and the original compiler:
>>>>
>>>> Fork, order-0, Apple M2:
>>>> | kernel                |   mean_rel |   std_rel |
>>>> |:----------------------|-----------:|----------:|
>>>> | mm-unstable           |       0.0% |      0.8% |
>>>> | hugetlb-rmap-cleanups |       1.3% |      2.0% |
>>>> | fork-batching         |       4.3% |      1.0% |
>>>>
>>>> Fork, order-9, Apple M2:
>>>> | kernel                |   mean_rel |   std_rel |
>>>> |:----------------------|-----------:|----------:|
>>>> | mm-unstable           |       0.0% |      0.8% |
>>>> | hugetlb-rmap-cleanups |       0.9% |      0.9% |
>>>> | fork-batching         |     -37.3% |      1.0% |
>>>>
>>>> Fork, order-0, Ampere Altra:
>>>> | kernel                |   mean_rel |   std_rel |
>>>> |:----------------------|-----------:|----------:|
>>>> | mm-unstable           |       0.0% |      0.7% |
>>>> | hugetlb-rmap-cleanups |       3.2% |      0.7% |
>>>> | fork-batching         |       5.5% |      1.1% |
>>>>
>>>> Fork, order-9, Ampere Altra:
>>>> | kernel                |   mean_rel |   std_rel |
>>>> |:----------------------|-----------:|----------:|
>>>> | mm-unstable           |       0.0% |      0.1% |
>>>> | hugetlb-rmap-cleanups |       0.5% |      0.1% |
>>>> | fork-batching         |     -10.4% |      0.1% |
>>>>
>>>
>>> I just gave it another quick benchmark run on that Intel system.
>>>
>>> hugetlb-rmap-cleanups -> fork-batching
>>>
>>> order-0: 0.014114 -> 0.013848
>>>
>>> -1.9%
>>>
>>> order-9: 0.014262 -> 0.009410
>>>
>>> -34%
>>>
>>> Note that I disable SMT and turbo, and pin the test to one CPU, to make the
>>> results as stable as possible. My kernel config has anything related to
>>> debugging disabled.
>>>
>>
>> And with gcc 13.2 on arm64:
>>
>> Fork, order-0, Apple M2 VM:
>> | kernel                |   mean_rel |   std_rel |
>> |:----------------------|-----------:|----------:|
>> | mm-unstable           |       0.0% |      1.5% |
>> | hugetlb-rmap-cleanups |      -3.3% |      1.1% |
>> | fork-batching         |      -3.6% |      1.4% |
>>
>> Fork, order-9, Apple M2 VM:
>> | kernel                |   mean_rel |   std_rel |
>> |:----------------------|-----------:|----------:|
>> | mm-unstable           |       0.0% |      1.8% |
>> | hugetlb-rmap-cleanups |      -5.8% |      1.3% |
>> | fork-batching         |     -38.1% |      2.3% |
>>
>> Fork, order-0, Ampere Altra:
>> | kernel                |   mean_rel |   std_rel |
>> |:----------------------|-----------:|----------:|
>> | mm-unstable           |       0.0% |      1.3% |
>> | hugetlb-rmap-cleanups |      -0.1% |      0.4% |
>> | fork-batching         |      -0.4% |      0.5% |
>>
>> Fork, order-9, Ampere Altra:
>> | kernel                |   mean_rel |   std_rel |
>> |:----------------------|-----------:|----------:|
>> | mm-unstable           |       0.0% |      0.1% |
>> | hugetlb-rmap-cleanups |      -0.1% |      0.1% |
>> | fork-batching         |     -13.9% |      0.1% |
>>
>>
>> So all looking good. Compiler was the issue. Sorry for the noise.
> 
> No need to be sorry, good that we figured out what's going wrong here.
> 
> Weird that the compiler makes such a difference here.
> 
>>
>> So please go ahead with you rmap v2 stuff, and I'll wait for you to post the
>> fork and zap batching patches properly, then rebase my arm64 contpte stuff on
>> top and remeasure everything.
> 
> Yes, will get rmap v2 out soon, then start working on fork, and then try
> tackling zap. I have some holiday coming up, so it might take some time -- but
> there is plenty of time left.

Me too, I'll be out from end of Friday, returning on 2nd Jan.

Happy Christmas!

>