Received: by 2002:a05:7412:d1aa:b0:fc:a2b0:25d7 with SMTP id ba42csp1671047rdb; Wed, 31 Jan 2024 05:58:23 -0800 (PST) X-Google-Smtp-Source: AGHT+IEWRckqs4AkVfq7ahqI1Tz/wThY5gJRtInx85LASPs5jKScQ1IvhavZ5Oerlmz/tadE4EH6 X-Received: by 2002:a05:6214:daf:b0:68c:5d50:1b32 with SMTP id h15-20020a0562140daf00b0068c5d501b32mr2110249qvh.47.1706709503283; Wed, 31 Jan 2024 05:58:23 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706709503; cv=pass; d=google.com; s=arc-20160816; b=WGQb79uKmVqztrOrJ0IlF7a5aqyl+SM5IoSECAoZBPe/fWyjTsybDv8tKT2j+9vy1d Z1w74EvtCzVeHIk8B6MEUp1bl5aqYC1xDd4EH0ogUgzQHvAg8492GdlsqvtL2G1ewyGs Kpf/WvJtbhYoCtP/wgPPVGnExSia0Vt7iNMUUKFBXxTT2nGWyk65G1ldK5iAR+6Oiv30 G44eBt+GaLdAxTqmVwfveHQehMCMJM4VMx60S23Hb1/ddZ1ux0NhLwBnLmd6bbwD0uj4 06BTKCTjjTsE0EltB/Ie+Tr4cf1NUR+Jv6XN453bhrVqavgZ30OltXKIvkuzu4qQI+3x dYgg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:date:message-id; bh=2QOBKqHmCcLmcnhqq2UxKVaPauOSH+hAEbvy5yWt3lM=; fh=yWEu2zr7H6ohA81FgwbzWL/e2dDJlPl4IfsBg8VF7Ic=; b=cJUpo+cTuGhVpB6D9dLkoFhFoLw6VocL1ZU3ZCGxH5nWHLXLGZqiZkm0+xJnzHaU3Z QS5WSZqRcdL/igktOqsTAhRb7X6AVpjtIy5tQ7qGUjysgEAZJ4kihgN3/KXi2A2e+0pH TuxZaJYKYOU4ikJNneCoDTYqm+YbsDU7y59f0VAn1FJOd3VjiTmOO0K3IpFUtdZfGXUr ZVD5NNVGpfOp9z4O7Dt7ceTbGIuN0yR386D37DKG+BD/GWdBb/nkDiqdRsSXkiHQ1pn3 TbaiZvL0ZntXTVRCfc+Bd50wgrKZ+3hAwFgFqZ+LKnKSz44iBhtNoumpUFrGJTZunvii Obgw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-46544-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-46544-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com X-Forwarded-Encrypted: i=1; AJvYcCVRwi2BKVTsWRlsJPkTKCu1K+tObe65ITrbUhPvGDtvGKfclp/bfMcsRWJWHWCkz9x4z7GjPCieSOO1dIhM6XZdAzBeKbjCeOurYcBzjg== Return-Path: Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id r9-20020a056214212900b0068562009ae9si5080870qvc.3.2024.01.31.05.58.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 05:58:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-46544-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-46544-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-46544-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id F218E1C21ECE for ; Wed, 31 Jan 2024 13:58:22 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7840D80BEB; Wed, 31 Jan 2024 13:58:15 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 772BA80BEF; Wed, 31 Jan 2024 13:58:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706709494; cv=none; b=obqN0V4virAtM89Evoofi4eb0y/Bq3GmIxmU0pdTewJUnGcqWM465P9BXLJwPijjIP/09HkGbTVRRygBMyCED3BpQRVkdogzTYna4YSy9Wo54btM3NXq3MsRTjWdjCdnsYz8+xIFremRDBdjSdr9gSZ3S6IHRe+eNIcxutXYfTU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706709494; c=relaxed/simple; bh=wXoTlE3Oqq9HkFer67OHjk568GGl78K+65liJbL6ueo=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=Ncs6Emhu8az+P2OneNa6/kWjlPhduzLdzvbYqYlCmXIfkvZLnmzplhs8zfm9M02PM8Vl5/XnioqAILK06MJdoWNilj6/IRUxrvxzVDH+kFKtzk1o51u7C6tmESQYhmaUBqVng79NlVbDOHLuT80+phy5LAMLbLMRBTss0t7zgEA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B8591DA7; Wed, 31 Jan 2024 05:58:54 -0800 (PST) Received: from [10.57.79.60] (unknown [10.57.79.60]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 724A83F738; Wed, 31 Jan 2024 05:58:06 -0800 (PST) Message-ID: Date: Wed, 31 Jan 2024 13:58:04 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 00/15] mm/memory: optimize fork() with PTE-mapped THP Content-Language: en-GB To: David Hildenbrand , linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, Andrew Morton , Matthew Wilcox , Russell King , Catalin Marinas , Will Deacon , Dinh Nguyen , Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , "David S. Miller" , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org References: <20240129124649.189745-1-david@redhat.com> <57eb82c7-4816-42a2-b5ab-cc221e289b21@arm.com> <714d0930-2202-48b6-9728-d248f820325e@arm.com> <30718fc8-15cf-41e4-922c-5cdbf00a0840@redhat.com> From: Ryan Roberts In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On 31/01/2024 13:38, David Hildenbrand wrote: >>>> Nope: looks the same. I've taken my test harness out of the picture and done >>>> everything manually from the ground up, with the old tests and the new. >>>> Headline >>>> is that I see similar numbers from both. >>> >>> I took me a while to get really reproducible numbers on Intel. Most importantly: >>> * Set a fixed CPU frequency, disabling any boost and avoiding any >>>    thermal throttling. >>> * Pin the test to CPUs and set a nice level. >> >> I'm already pinning the test to cpu 0. But for M2, at least, I'm running in a VM >> on top of macos, and I don't have a mechanism to pin the QEMU threads to the >> physical CPUs. Anyway, I don't think these are problems because for a given >> kernel build I can accurately repro numbers. > > Oh, you do have a layer of virtualization in there. I *suspect* that might > amplify some odd things regarding code layout, caching effects, etc. > > I guess especially the fork() benchmark is too sensible (fast) for things like > that, so I would just focus on bare metal results where you can control the > environment completely. Yeah, maybe. OK I'll park M2 for now. > > Note that regarding NUMA effects, I mean when some memory access within the same > socket is faster/slower even with only a single node. On AMD EPYC that's > possible, depending on which core you are running and on which memory controller > the memory you want to access is located. If both are in different quadrants > IIUC, the access latency will be different. I've configured the NUMA to only bring the RAM and CPUs for a single socket online, so I shouldn't be seeing any of these effects. Anyway, I've been using the Altra as a secondary because its so much slower than the M2. Let me move over to it and see if everything looks more straightforward there. > >>> But yes: I was observing something similar on AMD EPYC, where you get >>> consecutive pages from the buddy, but once you allocate from the PCP it might no >>> longer be consecutive. >>> >>>>    - test is 5-10% slower when output is printed to terminal vs when >>>> redirected to >>>>      file. I've always effectively been redirecting. Not sure if this overhead >>>>      could start to dominate the regression and that's why you don't see it? >>> >>> That's weird, because we don't print while measuring? Anyhow, 5/10% variance on >>> some system is not the end of the world. >> >> I imagine its cache effects? More work to do to print the output could be >> evicting some code that's in the benchmark path? > > Maybe. Do you also see these oddities on the bare metal system? > >> >>> >>>> >>>> I'm inclined to run this test for the last N kernel releases and if the number >>>> moves around significantly, conclude that these tests don't really matter. >>>> Otherwise its an exercise in randomly refactoring code until it works well, but >>>> that's just overfitting to the compiler and hw. What do you think? >>> >>> Personally, I wouldn't lose sleep if you see weird, unexplainable behavior on >>> some system (not even architecture!). Trying to optimize for that would indeed >>> be random refactorings. >>> >>> But I would not be so fast to say that "these tests don't really matter" and >>> then go wild and degrade them as much as you want. There are use cases that care >>> about fork performance especially with order-0 pages -- such as Redis. >> >> Indeed. But also remember that my fork baseline time is ~2.5ms, and I think you >> said yours was 14ms :) > > Yes, no idea why M2 is that fast (BTW, which page size? 4k or 16k? ) :) The guest kernel is using 4K pages. I'm not quite sure what is happening at stage2; QEMU doesn't expose any options to explicitly request huge pages for macos AFAICT. > >> >> I'll continue to mess around with it until the end of the day. But I'm not >> making any headway, then I'll change tack; I'll just measure the performance of >> my contpte changes using your fork/zap stuff as the baseline and post based on >> that. > > You should likely not focus on M2 results. Just pick a representative bare metal > machine where you get consistent, explainable results. > > Nothing in the code is fine-tuned for a particular architecture so far, only > order-0 handling is kept separate. > > BTW: I see the exact same speedups for dontneed that I see for munmap. For > example, for order-9, it goes from 0.023412s -> 0.009785, so -58%. So I'm > curious why you see a speedup for munmap but not for dontneed. Ugh... ok, coming up.