Received: by 2002:a6b:500f:0:0:0:0:0 with SMTP id e15csp4977071iob; Mon, 9 May 2022 06:10:11 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxCR4jtaEgAnYudhkHf9vyb9nITr9HFwB+rn9Hj9wbxyBAMGYLkDKz1x9Ro7/iS1XUz5MaU X-Received: by 2002:a63:db11:0:b0:3c5:ecce:6d8a with SMTP id e17-20020a63db11000000b003c5ecce6d8amr13053684pgg.550.1652101811451; Mon, 09 May 2022 06:10:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1652101811; cv=none; d=google.com; s=arc-20160816; b=khiDwer2nsppLc/rie/2gtebzVJICv4DLC9M0L1sFbSsNA8tza3sB1wKt8YQVPfHul mQAB9gTDAYU08QSj7KY7OAIASHpFShpX62HoDlAS5FoZsvowQ8lVOyFa1jsCxHw2A3l+ 4p8FSonhyv0soUTluq3y3MRtRATaR87uhUQ6jRH2KeCRIZTRqS2GKgqY6m9oxWTcHZfi jFs+AxETa7jTpY2sY/aVwx/P+TfqNvX2I/yc94j4C+DGCSrp5U6D/NU8E3UybUcSOePU OrkIL69d8e1AxhJPcWYWUWFvRxXQ0Sgw981XyG+H00M1KhhxBCMDYM5YfIeUzGBaUqk0 CzJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=5bT72D2LB3rMC2QzyyGp9UIfZaDKoeRpUeFqJ+hA8GI=; b=ZZHS9C4WrfdIP505foYReiLprIULV/ObxOg4ceVanKQC8XLATgMQ+IVEQWemBkXAVt 4EvspDLNbX9ZsyGtLugpw9FVJK1AxNBN+r9owg4nomSxQmqoaK+YgpfTBTeD29h+6d5e 96E/j0tU1BbL9nny3Kr98A8w7Nk42GI5ocgIrAedlJLRJabVyID38NhRlK4G8FpCrVU5 Juno8q96wj1GB5w5eneKW3TEFXNeF1AqEfoegxRWH/hd3gblyGzxS3RnTNL3HsoaoP/V 5Mrsq/jipbpdV3dTuybKIpF5mXAlq6nxFk7adkCBMBZuCokCIXA6Ww1M0LRsOvf/mHCB 7rcA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net. [2620:137:e000::1:18]) by mx.google.com with ESMTPS id c20-20020a631c54000000b003a9f9703091si14470508pgm.338.2022.05.09.06.10.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 May 2022 06:10:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 214CB2A7697; Mon, 9 May 2022 06:10:08 -0700 (PDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235292AbiEINNT (ORCPT + 99 others); Mon, 9 May 2022 09:13:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235274AbiEINNQ (ORCPT ); Mon, 9 May 2022 09:13:16 -0400 Received: from outbound-smtp15.blacknight.com (outbound-smtp15.blacknight.com [46.22.139.232]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8FA1A2944AD for ; Mon, 9 May 2022 06:09:21 -0700 (PDT) Received: from mail.blacknight.com (pemlinmail06.blacknight.ie [81.17.255.152]) by outbound-smtp15.blacknight.com (Postfix) with ESMTPS id E12A41C37E0 for ; Mon, 9 May 2022 14:09:19 +0100 (IST) Received: (qmail 19633 invoked from network); 9 May 2022 13:09:19 -0000 Received: from unknown (HELO morpheus.112glenside.lan) (mgorman@techsingularity.net@[84.203.198.246]) by 81.17.254.9 with ESMTPA; 9 May 2022 13:09:19 -0000 From: Mel Gorman To: Nicolas Saenz Julienne Cc: Marcelo Tosatti , Vlastimil Babka , Michal Hocko , LKML , Linux-MM , Mel Gorman Subject: [PATCH 6/6] mm/page_alloc: Remotely drain per-cpu lists Date: Mon, 9 May 2022 14:08:05 +0100 Message-Id: <20220509130805.20335-7-mgorman@techsingularity.net> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220509130805.20335-1-mgorman@techsingularity.net> References: <20220509130805.20335-1-mgorman@techsingularity.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RDNS_NONE, SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Nicolas Saenz Julienne Some setups, notably NOHZ_FULL CPUs, are too busy to handle the per-cpu drain work queued by __drain_all_pages(). So introduce new a mechanism to remotely drain the per-cpu lists. It is made possible by remotely locking 'struct per_cpu_pages' new per-cpu spinlocks. A benefit of this new scheme is that drain operations are now migration safe. There was no observed performance degradation vs. the previous scheme. Both netperf and hackbench were run in parallel to triggering the __drain_all_pages(NULL, true) code path around ~100 times per second. The new scheme performs a bit better (~5%), although the important point here is there are no performance regressions vs. the previous mechanism. Per-cpu lists draining happens only in slow paths. Link: https://lore.kernel.org/r/20211103170512.2745765-4-nsaenzju@redhat.com Signed-off-by: Nicolas Saenz Julienne Signed-off-by: Mel Gorman --- mm/page_alloc.c | 59 +++++-------------------------------------------- 1 file changed, 5 insertions(+), 54 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 7be21254087c..4ac39d30ec8f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -164,13 +164,7 @@ DEFINE_PER_CPU(int, _numa_mem_); /* Kernel "local memory" node */ EXPORT_PER_CPU_SYMBOL(_numa_mem_); #endif -/* work_structs for global per-cpu drains */ -struct pcpu_drain { - struct zone *zone; - struct work_struct work; -}; static DEFINE_MUTEX(pcpu_drain_mutex); -static DEFINE_PER_CPU(struct pcpu_drain, pcpu_drain); #ifdef CONFIG_GCC_PLUGIN_LATENT_ENTROPY volatile unsigned long latent_entropy __latent_entropy; @@ -3090,9 +3084,6 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, * Called from the vmstat counter updater to drain pagesets of this * currently executing processor on remote nodes after they have * expired. - * - * Note that this function must be called with the thread pinned to - * a single processor. */ void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp) { @@ -3117,10 +3108,6 @@ void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp) /* * Drain pcplists of the indicated processor and zone. - * - * The processor must either be the current processor and the - * thread pinned to the current processor or a processor that - * is not online. */ static void drain_pages_zone(unsigned int cpu, struct zone *zone) { @@ -3139,10 +3126,6 @@ static void drain_pages_zone(unsigned int cpu, struct zone *zone) /* * Drain pcplists of all zones on the indicated processor. - * - * The processor must either be the current processor and the - * thread pinned to the current processor or a processor that - * is not online. */ static void drain_pages(unsigned int cpu) { @@ -3155,9 +3138,6 @@ static void drain_pages(unsigned int cpu) /* * Spill all of this CPU's per-cpu pages back into the buddy allocator. - * - * The CPU has to be pinned. When zone parameter is non-NULL, spill just - * the single zone's pages. */ void drain_local_pages(struct zone *zone) { @@ -3169,24 +3149,6 @@ void drain_local_pages(struct zone *zone) drain_pages(cpu); } -static void drain_local_pages_wq(struct work_struct *work) -{ - struct pcpu_drain *drain; - - drain = container_of(work, struct pcpu_drain, work); - - /* - * drain_all_pages doesn't use proper cpu hotplug protection so - * we can race with cpu offline when the WQ can move this from - * a cpu pinned worker to an unbound one. We can operate on a different - * cpu which is alright but we also have to make sure to not move to - * a different one. - */ - migrate_disable(); - drain_local_pages(drain->zone); - migrate_enable(); -} - /* * The implementation of drain_all_pages(), exposing an extra parameter to * drain on all cpus. @@ -3207,13 +3169,6 @@ static void __drain_all_pages(struct zone *zone, bool force_all_cpus) */ static cpumask_t cpus_with_pcps; - /* - * Make sure nobody triggers this path before mm_percpu_wq is fully - * initialized. - */ - if (WARN_ON_ONCE(!mm_percpu_wq)) - return; - /* * Do not drain if one is already in progress unless it's specific to * a zone. Such callers are primarily CMA and memory hotplug and need @@ -3263,14 +3218,12 @@ static void __drain_all_pages(struct zone *zone, bool force_all_cpus) } for_each_cpu(cpu, &cpus_with_pcps) { - struct pcpu_drain *drain = per_cpu_ptr(&pcpu_drain, cpu); - - drain->zone = zone; - INIT_WORK(&drain->work, drain_local_pages_wq); - queue_work_on(cpu, mm_percpu_wq, &drain->work); + if (zone) { + drain_pages_zone(cpu, zone); + } else { + drain_pages(cpu); + } } - for_each_cpu(cpu, &cpus_with_pcps) - flush_work(&per_cpu_ptr(&pcpu_drain, cpu)->work); mutex_unlock(&pcpu_drain_mutex); } @@ -3279,8 +3232,6 @@ static void __drain_all_pages(struct zone *zone, bool force_all_cpus) * Spill all the per-cpu pages from all CPUs back into the buddy allocator. * * When zone parameter is non-NULL, spill just the single zone's pages. - * - * Note that this can be extremely slow as the draining happens in a workqueue. */ void drain_all_pages(struct zone *zone) { -- 2.34.1