Received: by 2002:a05:6a10:d5a5:0:0:0:0 with SMTP id gn37csp2147820pxb; Fri, 8 Oct 2021 01:42:47 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwOgceh2lWQHw3YTca1DvP4GuOyQz5NsVXNFhgf/86s0dVsi4OPTR+0/dUuSR/iJqc3DBsR X-Received: by 2002:a17:906:2a8f:: with SMTP id l15mr2824252eje.156.1633682567142; Fri, 08 Oct 2021 01:42:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1633682567; cv=none; d=google.com; s=arc-20160816; b=Lah1HFldbq+w/GXI5oDtm+ujCzV3+P8l8Q6sRKhyteKnbF+ipFSqh79vtsc/goa17x DYBeqnAGqCB3yNNVdYqocm24tnoeCiHTLanXZpqHwltbOoxQ8ru2CQpJ4Lc6uvNIXlcZ ImjHJGAyGQBOdq/X0NQiLO4paFYZTNeG2xDm7CoFRBRdMB0grotKLoHDnyIYwLMs3Kk3 83iQ+ghcpGSazEJ2VV/glBaYbkCdb4aXuS0EEBatNo3yYjDIgsHF0taSAC85eue336EW Zdo8LoXhhvWJDZCBGFpaIZA4aoC9V34Yh/tp3p4gf62wNn/6lK8MB1rYV8QiKa5ekk5l on8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=Ee7TAak6nAWWPcGovGoyPuKmw6nqGJGqqvG5dtvwwAo=; b=p9UmyWONlD9Pw4yUgqkboP4ml6pNYS2sWb/6PkGzCs3RQOPHKEkDWY6e0UB/RfhlQo dZWLBk7D/0UmX9SKpGPxoiv6APKCMALQuQoxa1E8cvS9qpXflMzU9jj//h2vxrCEoV+g cVgnS7M3ryPhu2AXdLh76s9QyO6OLRZgNUeMKpQvXhHQee73TX4uPrE1zxLkGW6qrsUA 75jSVMSY1tBV+Ny/fQLNG4Ybwb6oLCbTP5gUGmwQ5zK8xsLKR/Qw9PAySLGW2uERRIrS SWD/hN87j0GXTayXOKl3Vds32I96pJsK2ZWNdPwmEBVS108D2w+5UOYTe4CBOKlKqI+u krhg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id nd8si2738548ejc.657.2021.10.08.01.42.23; Fri, 08 Oct 2021 01:42:47 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236109AbhJHIma (ORCPT + 99 others); Fri, 8 Oct 2021 04:42:30 -0400 Received: from mga04.intel.com ([192.55.52.120]:42236 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234523AbhJHImM (ORCPT ); Fri, 8 Oct 2021 04:42:12 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10130"; a="225240240" X-IronPort-AV: E=Sophos;i="5.85,357,1624345200"; d="scan'208";a="225240240" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Oct 2021 01:40:17 -0700 X-IronPort-AV: E=Sophos;i="5.85,357,1624345200"; d="scan'208";a="439860409" Received: from yhuang6-desk2.sh.intel.com ([10.239.159.119]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Oct 2021 01:40:13 -0700 From: Huang Ying To: linux-kernel@vger.kernel.org Cc: Huang Ying , Andrew Morton , Michal Hocko , Rik van Riel , Mel Gorman , Peter Zijlstra , Dave Hansen , Yang Shi , Zi Yan , Wei Xu , osalvador , Shakeel Butt , linux-mm@kvack.org Subject: [PATCH -V9 6/6] memory tiering: adjust hot threshold automatically Date: Fri, 8 Oct 2021 16:39:38 +0800 Message-Id: <20211008083938.1702663-7-ying.huang@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211008083938.1702663-1-ying.huang@intel.com> References: <20211008083938.1702663-1-ying.huang@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org It isn't easy for the administrator to determine the hot threshold. So in this patch, a method to adjust the hot threshold automatically is implemented. The basic idea is to control the number of the candidate promotion pages to match the promotion rate limit. If the hint page fault latency of a page is less than the hot threshold, we will try to promote the page, and the page is called the candidate promotion page. If the number of the candidate promotion pages in the statistics interval is much more than the promotion rate limit, the hot threshold will be decreased to reduce the number of the candidate promotion pages. Otherwise, the hot threshold will be increased to increase the number of the candidate promotion pages. To make the above method works, in each statistics interval, the total number of the pages to check (on which the hint page faults occur) and the hot/cold distribution need to be stable. Because the page tables are scanned linearly in NUMA balancing, but the hot/cold distribution isn't uniform along the address, the statistics interval should be larger than the NUMA balancing scan period. So in the patch, the max scan period is used as statistics interval and it works well in our tests. The sysctl knob kernel.numa_balancing_hot_threshold_ms becomes the initial value and max value of the hot threshold. Signed-off-by: "Huang, Ying" Cc: Andrew Morton Cc: Michal Hocko Cc: Rik van Riel Cc: Mel Gorman Cc: Peter Zijlstra Cc: Dave Hansen Cc: Yang Shi Cc: Zi Yan Cc: Wei Xu Cc: osalvador Cc: Shakeel Butt Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org --- include/linux/mmzone.h | 3 ++ include/linux/sched/sysctl.h | 2 ++ kernel/sched/fair.c | 59 +++++++++++++++++++++++++++++++++--- kernel/sysctl.c | 3 +- 4 files changed, 62 insertions(+), 5 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index d6a0efd387bd..69bb672ea743 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -891,6 +891,9 @@ typedef struct pglist_data { #ifdef CONFIG_NUMA_BALANCING unsigned long numa_ts; unsigned long numa_nr_candidate; + unsigned long numa_threshold_ts; + unsigned long numa_threshold_nr_candidate; + unsigned long numa_threshold; #endif /* Fields commonly accessed by the page reclaim scanner */ diff --git a/include/linux/sched/sysctl.h b/include/linux/sched/sysctl.h index 7d937adaac0f..ff2c43e8ebac 100644 --- a/include/linux/sched/sysctl.h +++ b/include/linux/sched/sysctl.h @@ -84,6 +84,8 @@ int sysctl_sched_uclamp_handler(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos); int sysctl_numa_balancing(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos); +int sysctl_numa_balancing_threshold(struct ctl_table *table, int write, void *buffer, + size_t *lenp, loff_t *ppos); int sysctl_schedstats(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index c57baeacfc1a..ff57055aab23 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1465,6 +1465,54 @@ static bool numa_migration_check_rate_limit(struct pglist_data *pgdat, return true; } +int sysctl_numa_balancing_threshold(struct ctl_table *table, int write, void *buffer, + size_t *lenp, loff_t *ppos) +{ + int err; + struct pglist_data *pgdat; + + if (write && !capable(CAP_SYS_ADMIN)) + return -EPERM; + + err = proc_dointvec_minmax(table, write, buffer, lenp, ppos); + if (err < 0 || !write) + return err; + + for_each_online_pgdat(pgdat) + pgdat->numa_threshold = 0; + + return err; +} + +#define NUMA_MIGRATION_ADJUST_STEPS 16 + +static void numa_migration_adjust_threshold(struct pglist_data *pgdat, + unsigned long rate_limit, + unsigned long ref_th) +{ + unsigned long now = jiffies, last_th_ts, th_period; + unsigned long unit_th, th; + unsigned long nr_cand, ref_cand, diff_cand; + + th_period = msecs_to_jiffies(sysctl_numa_balancing_scan_period_max); + last_th_ts = pgdat->numa_threshold_ts; + if (now > last_th_ts + th_period && + cmpxchg(&pgdat->numa_threshold_ts, last_th_ts, now) == last_th_ts) { + ref_cand = rate_limit * + sysctl_numa_balancing_scan_period_max / 1000; + nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE); + diff_cand = nr_cand - pgdat->numa_threshold_nr_candidate; + unit_th = ref_th / NUMA_MIGRATION_ADJUST_STEPS; + th = pgdat->numa_threshold ? : ref_th; + if (diff_cand > ref_cand * 11 / 10) + th = max(th - unit_th, unit_th); + else if (diff_cand < ref_cand * 9 / 10) + th = min(th + unit_th, ref_th); + pgdat->numa_threshold_nr_candidate = nr_cand; + pgdat->numa_threshold = th; + } +} + bool should_numa_migrate_memory(struct task_struct *p, struct page * page, int src_nid, int dst_cpu) { @@ -1479,19 +1527,22 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page, if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && !node_is_toptier(src_nid)) { struct pglist_data *pgdat; - unsigned long rate_limit, latency, th; + unsigned long rate_limit, latency, th, def_th; pgdat = NODE_DATA(dst_nid); if (pgdat_free_space_enough(pgdat)) return true; - th = sysctl_numa_balancing_hot_threshold; + def_th = sysctl_numa_balancing_hot_threshold; + rate_limit = + sysctl_numa_balancing_rate_limit << (20 - PAGE_SHIFT); + numa_migration_adjust_threshold(pgdat, rate_limit, def_th); + + th = pgdat->numa_threshold ? : def_th; latency = numa_hint_fault_latency(page); if (latency > th) return false; - rate_limit = - sysctl_numa_balancing_rate_limit << (20 - PAGE_SHIFT); return numa_migration_check_rate_limit(pgdat, rate_limit, thp_nr_pages(page)); } diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 0d89021bd66a..0a87d5877718 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -1816,7 +1816,8 @@ static struct ctl_table kern_table[] = { .data = &sysctl_numa_balancing_hot_threshold, .maxlen = sizeof(unsigned int), .mode = 0644, - .proc_handler = proc_dointvec, + .proc_handler = sysctl_numa_balancing_threshold, + .extra1 = SYSCTL_ZERO, }, { .procname = "numa_balancing_rate_limit_mbps", -- 2.30.2