Received-SPF: pass (google.com: domain of linux-kernel+bounces-46687-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <eeafba09-058f-47da-bbb9-19ded5ad05da@arm.com>
Date: Wed, 31 Jan 2024 15:08:58 +0000
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v3 00/15] mm/memory: optimize fork() with PTE-mapped THP
Content-Language: en-GB
To: David Hildenbrand <david@redhat.com>, linux-kernel@vger.kernel.org
Cc: linux-mm@kvack.org, Andrew Morton <akpm@linux-foundation.org>,
 Matthew Wilcox <willy@infradead.org>, Russell King <linux@armlinux.org.uk>,
 Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>,
 Dinh Nguyen <dinguyen@kernel.org>, Michael Ellerman <mpe@ellerman.id.au>,
 Nicholas Piggin <npiggin@gmail.com>,
 Christophe Leroy <christophe.leroy@csgroup.eu>,
 "Aneesh Kumar K.V" <aneesh.kumar@kernel.org>,
 "Naveen N. Rao" <naveen.n.rao@linux.ibm.com>,
 Paul Walmsley <paul.walmsley@sifive.com>, Palmer Dabbelt
 <palmer@dabbelt.com>, Albert Ou <aou@eecs.berkeley.edu>,
 Alexander Gordeev <agordeev@linux.ibm.com>,
 Gerald Schaefer <gerald.schaefer@linux.ibm.com>,
 Heiko Carstens <hca@linux.ibm.com>, Vasily Gorbik <gor@linux.ibm.com>,
 Christian Borntraeger <borntraeger@linux.ibm.com>,
 Sven Schnelle <svens@linux.ibm.com>, "David S. Miller"
 <davem@davemloft.net>, linux-arm-kernel@lists.infradead.org,
 linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org,
 linux-s390@vger.kernel.org, sparclinux@vger.kernel.org
References: <20240129124649.189745-1-david@redhat.com>
 <a335a9d2-9b8f-4eb8-ba22-23a223b59b06@arm.com>
 <a1a0e9b3-dae2-418f-bd63-50e65f471728@redhat.com>
 <57eb82c7-4816-42a2-b5ab-cc221e289b21@arm.com>
 <e6eaba5b-f290-4d1f-990b-a47d89f56ee4@redhat.com>
 <714d0930-2202-48b6-9728-d248f820325e@arm.com>
 <dcaa20c4-bd1f-4f15-bb0a-3a790808937d@arm.com>
 <30718fc8-15cf-41e4-922c-5cdbf00a0840@redhat.com>
 <de975655-8f8f-40dc-b281-75c40dd1e2c1@arm.com>
 <c63870b0-690a-4051-b4f5-296cf3b73be2@redhat.com>
 <a0cdeb7c-dec8-4971-8b54-e6f65ea48ade@arm.com>
 <74333154-a99b-4bad-81f4-bee02ba05e91@redhat.com>
 <a34eee7e-3970-4cdd-8c09-bca51132db50@arm.com>
 <cee2c0ed-661d-4948-8bc9-77c87c486c86@redhat.com>
From: Ryan Roberts <ryan.roberts@arm.com>
In-Reply-To: <cee2c0ed-661d-4948-8bc9-77c87c486c86@redhat.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

On 31/01/2024 15:05, David Hildenbrand wrote:
> On 31.01.24 16:02, Ryan Roberts wrote:
>> On 31/01/2024 14:29, David Hildenbrand wrote:
>>>>> Note that regarding NUMA effects, I mean when some memory access within the
>>>>> same
>>>>> socket is faster/slower even with only a single node. On AMD EPYC that's
>>>>> possible, depending on which core you are running and on which memory
>>>>> controller
>>>>> the memory you want to access is located. If both are in different quadrants
>>>>> IIUC, the access latency will be different.
>>>>
>>>> I've configured the NUMA to only bring the RAM and CPUs for a single socket
>>>> online, so I shouldn't be seeing any of these effects. Anyway, I've been using
>>>> the Altra as a secondary because its so much slower than the M2. Let me move
>>>> over to it and see if everything looks more straightforward there.
>>>
>>> Better use a system where people will actually run Linux production workloads
>>> on, even if it is slower :)
>>>
>>> [...]
>>>
>>>>>>
>>>>>> I'll continue to mess around with it until the end of the day. But I'm not
>>>>>> making any headway, then I'll change tack; I'll just measure the
>>>>>> performance of
>>>>>> my contpte changes using your fork/zap stuff as the baseline and post
>>>>>> based on
>>>>>> that.
>>>>>
>>>>> You should likely not focus on M2 results. Just pick a representative bare
>>>>> metal
>>>>> machine where you get consistent, explainable results.
>>>>>
>>>>> Nothing in the code is fine-tuned for a particular architecture so far, only
>>>>> order-0 handling is kept separate.
>>>>>
>>>>> BTW: I see the exact same speedups for dontneed that I see for munmap. For
>>>>> example, for order-9, it goes from 0.023412s -> 0.009785, so -58%. So I'm
>>>>> curious why you see a speedup for munmap but not for dontneed.
>>>>
>>>> Ugh... ok, coming up.
>>>
>>> Hopefully you were just staring at the wrong numbers (e.g., only with fork
>>> patches). Because both (munmap/pte-dontneed) are using the exact same code path.
>>>
>>
>> Ahh... I'm doing pte-dontneed, which is the only option in your original
>> benchmark - it does MADV_DONTNEED one page at a time. It looks like your new
>> benchmark has an additional "dontneed" option that does it in one shot. Which
>> option are you running? Assuming the latter, I think that explains it.
> 
> I temporarily removed that option and then re-added it. Guess you got a wrong
> snapshot of the benchmark :D
> 
> pte-dontneed not observing any change is great (no batching possible).

indeed.

> 
> dontneed should hopefully/likely see a speedup.

Yes, but that's almost exactly the same path as munmap, so I'm sure it really
adds much for this particular series. Anyway, on Altra at least, I'm seeing no
regressions, so:

Tested-by: Ryan Roberts <ryan.roberts@arm.com>

> 
> Great!
>