Received: by 2002:a05:7412:419a:b0:f3:1519:9f41 with SMTP id i26csp440551rdh; Thu, 23 Nov 2023 08:01:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IE6angkY2MQc8FPpLYIELUNvp8Zo5jj5G9T/PhF870i6K8QFoBjDllDhdLI7OfktFU9hTCW X-Received: by 2002:a05:6808:f11:b0:3b8:4125:c1e with SMTP id m17-20020a0568080f1100b003b841250c1emr5709252oiw.31.1700755305111; Thu, 23 Nov 2023 08:01:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700755305; cv=none; d=google.com; s=arc-20160816; b=iQ7CX6x/RZKG0Ci8OKNwkdLyV6c2JhrzJvVUhckmkKjhU8z/2y7fXXMm6Ur76LMmln 1qJ+VE+QOaXG6NrXjzjD8siD50LqqOa88dVjqhhe3ixK3L91O0D2kbuZ+FtGlzQurRmx xQjL8YUYvQ4rWs49VbjjZdk/4dAq3942qqttmrm1OUIION/T5Rsyibv0aSoUFIRNNM73 O1RpVe0x4/nPO1K/eZuRwX32UyVHNzDvMKcst01tXdAL2+3R/9ZlCopkzEV2XdF8hscU Rl6yOFAYylvVQXgdx0hNAkhTxIpAD/bzooXakHKkcf6Pe1IbdyolSl+kAUgvX8ktI9e3 11Uw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:in-reply-to:from :references:cc:to:content-language:subject:user-agent:mime-version :date:message-id; bh=sTG9jHMZ3sHydKvODMAnIczrL3GVmZgL5IHAwRQ2jjo=; fh=Om0/u7nXBQ4LaIxc2I5LOM+DE9wxQWy9xmDIpiEl7pg=; b=kkbSwpbeMs+ZZrA/X7BkNtF/Blw3drP+4JO3URZxf2VoijNH/klQo9Y2v3/S77PWRv Qa3x9OmxMaWMJXlcKA8rwTGTyluA23RuzbsNZAB/jnbDdfa9QR4An1i6REEAS2ABIgy9 yFr3sUS46ZZoK5/npCJIL1Kn/3GqlhgwB7aMQnIg4qUAMuLQKX0S7OP2WS4HQQ8PNsmr YWho1FIps4w+e/jJEprn2gfbxbTRwfnDnDU+ExFW9ZpecNxD3FmYA+pIN5PpRtWcRHmi l4MUJCbzNz57a1X+nHjHg94pTjbXkn0yDPSohRzL+nz2Y00xahBdNvuI9blhNMXTitjL rgIA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Return-Path: Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id gu14-20020a0568082f0e00b003b2f2b52416si618302oib.214.2023.11.23.08.01.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Nov 2023 08:01:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 4351C81EA706; Thu, 23 Nov 2023 08:01:42 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229731AbjKWQBZ (ORCPT + 99 others); Thu, 23 Nov 2023 11:01:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229698AbjKWQBX (ORCPT ); Thu, 23 Nov 2023 11:01:23 -0500 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id ABEDB83 for ; Thu, 23 Nov 2023 08:01:29 -0800 (PST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CDB3312FC; Thu, 23 Nov 2023 08:02:15 -0800 (PST) Received: from [10.1.37.168] (XHFQ2J9959.cambridge.arm.com [10.1.37.168]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 48E003F6C4; Thu, 23 Nov 2023 08:01:26 -0800 (PST) Message-ID: <3b4f6bff-6322-4394-9efb-9c3b9ef52010@arm.com> Date: Thu, 23 Nov 2023 16:01:25 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 14/14] arm64/mm: Add ptep_get_and_clear_full() to optimize process teardown Content-Language: en-GB To: Alistair Popple Cc: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20231115163018.1303287-1-ryan.roberts@arm.com> <20231115163018.1303287-15-ryan.roberts@arm.com> <87fs0xxd5g.fsf@nvdebian.thelocal> From: Ryan Roberts In-Reply-To: <87fs0xxd5g.fsf@nvdebian.thelocal> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Thu, 23 Nov 2023 08:01:42 -0800 (PST) On 23/11/2023 05:13, Alistair Popple wrote: > > Ryan Roberts writes: > >> ptep_get_and_clear_full() adds a 'full' parameter which is not present >> for the fallback ptep_get_and_clear() function. 'full' is set to 1 when >> a full address space teardown is in progress. We use this information to >> optimize arm64_sys_exit_group() by avoiding unfolding (and therefore >> tlbi) contiguous ranges. Instead we just clear the PTE but allow all the >> contiguous neighbours to keep their contig bit set, because we know we >> are about to clear the rest too. >> >> Before this optimization, the cost of arm64_sys_exit_group() exploded to >> 32x what it was before PTE_CONT support was wired up, when compiling the >> kernel. With this optimization in place, we are back down to the >> original cost. >> >> This approach is not perfect though, as for the duration between >> returning from the first call to ptep_get_and_clear_full() and making >> the final call, the contpte block in an intermediate state, where some >> ptes are cleared and others are still set with the PTE_CONT bit. If any >> other APIs are called for the ptes in the contpte block during that >> time, we have to be very careful. The core code currently interleaves >> calls to ptep_get_and_clear_full() with ptep_get() and so ptep_get() >> must be careful to ignore the cleared entries when accumulating the >> access and dirty bits - the same goes for ptep_get_lockless(). The only >> other calls we might resonably expect are to set markers in the >> previously cleared ptes. (We shouldn't see valid entries being set until >> after the tlbi, at which point we are no longer in the intermediate >> state). Since markers are not valid, this is safe; set_ptes() will see >> the old, invalid entry and will not attempt to unfold. And the new pte >> is also invalid so it won't attempt to fold. We shouldn't see this for >> the 'full' case anyway. >> >> The last remaining issue is returning the access/dirty bits. That info >> could be present in any of the ptes in the contpte block. ptep_get() >> will gather those bits from across the contpte block. We don't bother >> doing that here, because we know that the information is used by the >> core-mm to mark the underlying folio as accessed/dirty. And since the >> same folio must be underpinning the whole block (that was a requirement >> for folding in the first place), that information will make it to the >> folio eventually once all the ptes have been cleared. This approach >> means we don't have to play games with accumulating and storing the >> bits. It does mean that any interleaved calls to ptep_get() may lack >> correct access/dirty information if we have already cleared the pte that >> happened to store it. The core code does not rely on this though. > > Does not *currently* rely on this. I can't help but think it is > potentially something that could change in the future though which would > lead to some subtle bugs. Yes, there is a risk, although IMHO, its very small. > > Would there be any may of avoiding this? Half baked thought but could > you for example copy the access/dirty information to the last (or > perhaps first, most likely invalid) PTE? I spent a long time thinking about this and came up with a number of possibilities, none of them ideal. In the end, I went for the simplest one (which works but suffers from the problem that it depends on the way it is called not changing). 1) copy the access/dirty flags into all the remaining uncleared ptes within the contpte block. This is how I did it in v1; although it was racy. I think this could be implemented correctly but its extremely complex. 2) batch calls from the core-mm (like I did for pte_set_wrprotects()) so that we can clear 1 or more full contpte blocks in a single call - the ptes are never in an intermediate state. This is difficult because ptep_get_and_clear_full() returns the pte that was cleared so its difficult to scale that up to multiple ptes. 3) add ptep_get_no_access_dirty() and redefine the interface to only allow that to be called while ptep_get_and_clear_full() calls are on-going. Then assert in the other functions that ptep_get_and_clear_full() is not on-going when they are called. So we would get a clear sign that usage patterns have changed. But there is no easy place to store that state (other than scanning a contpte block looking for pte_none() amongst pte_valid_cont() entries) and it all felt ugly. 4) The simple approach I ended up taking; I thought it would be best to keep it simple and see if anyone was concerned before doing something more drastic. What do you think? If we really need to solve this, then option 1 is my preferred route, but it would take some time to figure out and reason about a race-free scheme. Thanks, Ryan