Received: by 2002:a05:7412:bb8d:b0:d7:7d3a:4fe2 with SMTP id js13csp1135500rdb; Wed, 16 Aug 2023 02:51:25 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFD2s2LlH+G6FwDwSK61WYUdsJrtIW6OhYUWGPbLfcl5vbKEyRW57gZCAREoXFIMHc06OgU X-Received: by 2002:a17:906:6a22:b0:99c:d069:d1e with SMTP id qw34-20020a1709066a2200b0099cd0690d1emr1178428ejc.46.1692179484958; Wed, 16 Aug 2023 02:51:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1692179484; cv=none; d=google.com; s=arc-20160816; b=coWryjmqYXAsixeJi7QEBV7UxgRk4Zp958sckt2A+AEh5vvjHoIS6Q2bSMNSi/T7kH qaKGkinUZrsdCtOSuLRy9sLe/IMJ2iagDxm3JisG8nduVlKnwvRRqCELEROgKJxdOdyo 8hB0PD/KK79nHW705p3oPODUZKPKjesdAi3G2dVSABcb4w24jg/VEF2S5qopsjXR+6XQ GRthPPGOx4PQndOYwHx3xnz3eMFaaI/XUMGvhSDU1ahzwTmuzvgVaRq+1P3LsEOVA9wD ySogL6XQCJxkVVBm484fn72IfgLFSiXz55W/Q8V4PRxYI7gnNYZWSRXrBktEbWvua0ta QTng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:user-agent:in-reply-to:content-disposition :mime-version:references:message-id:subject:cc:to:from:date; bh=Zst07R4YFgAbHvvVjVlFmbChd1iDRXKS4lv4OFoDAdw=; fh=jaUHfWk/HIwS2KQHQWYF7RZzHkoF9NSQttMPykLXMc4=; b=ohgnELiOpSXwHKkwmlvkpuPcrltVNNHMVUceOe/q+SmdNp7PrBs+8mWMtaV1eVrW65 01Y0TSfXtB/pMdkCsv+NfJqvj6+F4XV6fMzifnD2ViCzGhjav7GXMrRY51uWZgcd9Z// A1BPUO2s1dfbVoWlUNgUXQFn2FfTZV/AmeqheJTSCbUVeg/qpMYTBCwXa3arsXytQwZj BteD4KdwW2ev8h6yEYoQrf0ryRfIjKCwq1+RzMVGYv36jC0g7J8CCtmIlRQ2Z6rT7Efh Il8BSihCByR+a/0+ZnGTuGKlQmpp4YLJFM+ssCF8i78Yi32zxofKyWbR0ae+IHmSv8EK 3hNw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o16-20020a17090608d000b00965a6d6b536si10038722eje.335.2023.08.16.02.50.56; Wed, 16 Aug 2023 02:51:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240765AbjHPAQK (ORCPT + 99 others); Tue, 15 Aug 2023 20:16:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240763AbjHPAQJ (ORCPT ); Tue, 15 Aug 2023 20:16:09 -0400 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 167C71986 for ; Tue, 15 Aug 2023 17:16:05 -0700 (PDT) X-AuditID: a67dfc5b-d85ff70000001748-f0-64dc1542ce1b Date: Wed, 16 Aug 2023 09:13:07 +0900 From: Byungchul Park To: "Huang, Ying" Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel_team@skhynix.com, akpm@linux-foundation.org, namit@vmware.com, xhao@linux.alibaba.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, dave.hansen@linux.intel.com Subject: Re: [RFC 2/2] mm: Defer TLB flush by keeping both src and dst folios at migration Message-ID: <20230816001307.GA44941@system.software.com> References: <20230804061850.21498-1-byungchul@sk.com> <20230804061850.21498-3-byungchul@sk.com> <877cpx9jsx.fsf@yhuang6-desk2.ccr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <877cpx9jsx.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Mutt/1.9.4 (2018-02-28) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprEIsWRmVeSWpSXmKPExsXC9ZZnoa6T6J0Ug2t7NCzmrF/DZvFiQzuj xdf1v5gtnn7qY7G4vGsOm8W9Nf9ZLc7vWstqsWPpPiaL67seMloc7z3AZPH7B1B2zhQri5Oz JrM48Hos2FTqsXmFlsfiPS+ZPDat6mTz2PRpErvHiRm/WTx2PrT0mHcy0OP9vqtsHlt/2Xl8 3iTn8W7+W7YAnigum5TUnMyy1CJ9uwSujFW3N7AWtOlUPJ5/ibGBca1iFyMnh4SAicSMc0uY YOzHM5vZQWwWAVWJo+1NLCA2m4C6xI0bP5lBbBEBDYlPC5eD1TALbGOSWHDPAcQWFoiROLHl M9gcXgELiT0tj4BsLg4hgamMEnPmbWGGSAhKnJz5hAWiWUvixr+XQEUcQLa0xPJ/HCBhTgE7 ibULL4CViwooSxzYdhxsjoTAfzaJ2Zf3sUIcKilxcMUNlgmMArOQjJ2FZOwshLELGJlXMQpl 5pXlJmbmmOhlVOZlVugl5+duYgTG1LLaP9E7GD9dCD7EKMDBqMTDy7DwdooQa2JZcWXuIUYJ DmYlEd4e3lspQrwpiZVVqUX58UWlOanFhxilOViUxHmNvpWnCAmkJ5akZqemFqQWwWSZODil GhglnAoOSRS3Hw59fiLtZOEJI6vdf/fJuqnwX7jxcsY9j9XsuyrWdF3q83eYt3HaI6WZN5xX Ly3x9zFxqT33aUEG0zZOv/abK9Y96J+5bj57bK6xcqcYf2zMjtr8mRtfCHuXfgg02XJxevXP yjiTySuPLX8gHLer+5z+RAbuT2l3ohak/2z6H3VXiaU4I9FQi7moOBEA+lBfq6UCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrJLMWRmVeSWpSXmKPExsXC5WfdrOskeifF4MIHCYs569ewWbzY0M5o 8XX9L2aLp5/6WCwOzz3JanF51xw2i3tr/rNanN+1ltVix9J9TBbXdz1ktDjee4DJ4vcPoOyc KVYWJ2dNZnHg81iwqdRj8wotj8V7XjJ5bFrVyeax6dMkdo8TM36zeOx8aOkx72Sgx/t9V9k8 Fr/4wOSx9Zedx+dNch7v5r9lC+CN4rJJSc3JLEst0rdL4MpYdXsDa0GbTsXj+ZcYGxjXKnYx cnJICJhIPJ7ZzA5iswioShxtb2IBsdkE1CVu3PjJDGKLCGhIfFq4HKyGWWAbk8SCew4gtrBA jMSJLZ+ZQGxeAQuJPS2PgGwuDiGBqYwSc+ZtYYZICEqcnPmEBaJZS+LGv5dARRxAtrTE8n8c IGFOATuJtQsvgJWLCihLHNh2nGkCI+8sJN2zkHTPQuhewMi8ilEkM68sNzEzx1SvODujMi+z Qi85P3cTIzBCltX+mbiD8ctl90OMAhyMSjy8DAtvpwixJpYVV+YeYpTgYFYS4e3hvZUixJuS WFmVWpQfX1Sak1p8iFGag0VJnNcrPDVBSCA9sSQ1OzW1ILUIJsvEwSnVwOh7Iel8tO2HlfWi Uk2NvsIXPrREsVd2vxe6484dtrTF9aGM74Zarc+Ss9o4KidkNvIVu1nqJPd6xL836Z72h3/t 6/3yupLnTk2WSme0fnpT8seud0c9tsc/tFi5j/fAjqRopR2PzuYnP7FYzbZf6qaGS/bzNRsz 30S2dxt/eGky8cLpxvQ9NUosxRmJhlrMRcWJAOfD40uMAgAA X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Aug 15, 2023 at 09:27:26AM +0800, Huang, Ying wrote: > Byungchul Park writes: > > > Implementation of CONFIG_MIGRC that stands for 'Migration Read Copy'. > > > > We always face the migration overhead at either promotion or demotion, > > while working with tiered memory e.g. CXL memory and found out TLB > > shootdown is a quite big one that is needed to get rid of if possible. > > > > Fortunately, TLB flush can be defered or even skipped if both source and > > destination of folios during migration are kept until all TLB flushes > > required will have been done, of course, only if the target PTE entries > > have read only permission, more precisely speaking, don't have write > > permission. Otherwise, no doubt the folio might get messed up. > > > > To achieve that: > > > > 1. For the folios that have only non-writable TLB entries, prevent > > TLB flush by keeping both source and destination of folios during > > migration, which will be handled later at a better time. > > > > 2. When any non-writable TLB entry changes to writable e.g. through > > fault handler, give up CONFIG_MIGRC mechanism so as to perform > > TLB flush required right away. > > > > 3. TLB flushes can be skipped if all TLB flushes required to free the > > duplicated folios have been done by any reason, which doesn't have > > to be done from migrations. > > > > 4. Adjust watermark check routine, __zone_watermark_ok(), with the > > number of duplicated folios because those folios can be freed > > and obtained right away through appropreate TLB flushes. > > > > 5. Perform TLB flushes and free the duplicated folios pending the > > flushes if page allocation routine is in trouble due to memory > > pressure, even more aggresively for high order allocation. > > Is the optimization restricted for page migration only? Can it be used > for other places? Like page reclaiming? Just to make sure, are you talking about the (5) description? For now, it's performed at the beginning of __alloc_pages_slowpath(), say, before page recaiming. Do you think it'd be meaningful to perform it during page reclaiming? Or do you mean something else? > > The measurement result: > > > > Architecture - x86_64 > > QEMU - kvm enabled, host cpu, 2nodes((4cpus, 2GB)+(cpuless, 6GB)) > > Linux Kernel - v6.4, numa balancing tiering on, demotion enabled > > Benchmark - XSBench with no parameter changed > > > > run 'perf stat' using events: > > (FYI, process wide result ~= system wide result(-a option)) > > 1) itlb.itlb_flush > > 2) tlb_flush.dtlb_thread > > 3) tlb_flush.stlb_any > > > > run 'cat /proc/vmstat' and pick up: > > 1) pgdemote_kswapd > > 2) numa_pages_migrated > > 3) pgmigrate_success > > 4) nr_tlb_remote_flush > > 5) nr_tlb_remote_flush_received > > 6) nr_tlb_local_flush_all > > 7) nr_tlb_local_flush_one > > > > BEFORE - mainline v6.4 > > ========================================== > > > > $ perf stat -e itlb.itlb_flush,tlb_flush.dtlb_thread,tlb_flush.stlb_any ./XSBench > > > > Performance counter stats for './XSBench': > > > > 426856 itlb.itlb_flush > > 6900414 tlb_flush.dtlb_thread > > 7303137 tlb_flush.stlb_any > > > > 33.500486566 seconds time elapsed > > 92.852128000 seconds user > > 10.526718000 seconds sys > > > > $ cat /proc/vmstat > > > > ... > > pgdemote_kswapd 1052596 > > numa_pages_migrated 1052359 > > pgmigrate_success 2161846 > > nr_tlb_remote_flush 72370 > > nr_tlb_remote_flush_received 213711 > > nr_tlb_local_flush_all 3385 > > nr_tlb_local_flush_one 198679 > > ... > > > > AFTER - mainline v6.4 + CONFIG_MIGRC > > ========================================== > > > > $ perf stat -e itlb.itlb_flush,tlb_flush.dtlb_thread,tlb_flush.stlb_any ./XSBench > > > > Performance counter stats for './XSBench': > > > > 179537 itlb.itlb_flush > > 6131135 tlb_flush.dtlb_thread > > 6920979 tlb_flush.stlb_any > > It appears that the number of "itlb.itlb_flush" changes much, but not > for other 2 events. Because the text segment of the executable file is > mapped as read-only? And most other pages are mapped read-write? Yes, for this benchmarch, XSBench. I didn't noticed that until checking it using perf event either. > > 30.396700625 seconds time elapsed > > 80.331252000 seconds user > > 10.303761000 seconds sys > > > > $ cat /proc/vmstat > > > > ... > > pgdemote_kswapd 1044602 > > numa_pages_migrated 1044202 > > pgmigrate_success 2157808 > > nr_tlb_remote_flush 30453 > > nr_tlb_remote_flush_received 88840 > > nr_tlb_local_flush_all 3039 > > nr_tlb_local_flush_one 198875 > > ... > > > > Signed-off-by: Byungchul Park [...] > > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > > index 306a3d1a0fa6..3be66d3eabd2 100644 > > --- a/include/linux/mm_types.h > > +++ b/include/linux/mm_types.h > > @@ -228,6 +228,10 @@ struct page { > > #ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS > > int _last_cpupid; > > #endif > > +#ifdef CONFIG_MIGRC > > + struct llist_node migrc_node; > > + unsigned int migrc_state; > > +#endif > > We cannot enlarge "struct page". This is what I worried about. Do you have a better idea? I don't think they fit onto page_ext or something. Byungchul