Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753426AbdHPDD5 (ORCPT ); Tue, 15 Aug 2017 23:03:57 -0400 Received: from mga11.intel.com ([192.55.52.93]:12576 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752130AbdHPDDx (ORCPT ); Tue, 15 Aug 2017 23:03:53 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.41,380,1498546800"; d="scan'208";a="300657236" Subject: Re: [PATCH 2/2] mm: Update NUMA counter threshold size To: Tim Chen , Mel Gorman Cc: Andrew Morton , Michal Hocko , Johannes Weiner , Dave , Andi Kleen , Jesper Dangaard Brouer , Ying Huang , Aaron Lu , Tim Chen , Linux MM , Linux Kernel References: <1502786736-21585-1-git-send-email-kemi.wang@intel.com> <1502786736-21585-3-git-send-email-kemi.wang@intel.com> <20170815095819.5kjh4rrhkye3lgf2@techsingularity.net> From: kemi Message-ID: Date: Wed, 16 Aug 2017 11:02:42 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2368 Lines: 54 On 2017年08月16日 00:55, Tim Chen wrote: > On 08/15/2017 02:58 AM, Mel Gorman wrote: >> On Tue, Aug 15, 2017 at 04:45:36PM +0800, Kemi Wang wrote: >> I'm fairly sure this pushes the size of that structure into the next >> cache line which is not welcome. >>>> vm_numa_stat_diff is an always incrementing field. How much do you gain >> if this becomes a u8 code and remove any code that deals with negative >> values? That would double the threshold without consuming another cache line. > > Doubling the threshold and counter size will help, but not as much > as making them above u8 limit as seen in Kemi's data: > > 125 537 358906028 <==> system by default (base) > 256 468 412397590 > 32765 394(-26.6%) 488932078(+36.2%) <==> with this patchset > > For small system making them u8 makes sense. For larger ones the > frequent local counter overflow into the global counter still > causes a lot of cache bounce. Kemi can perhaps collect some data > to see what is the gain from making the counters u8. > Tim, thanks for your answer. That is what I want to clarify. Also, pls notice that the negative threshold/2 is set to cpu local counter (e.g. vm_numa_stat_diff[]) once per-zone counter is updated in current code path. This weakens the benefit of changing s8 to u8 in this case. >> >> Furthermore, the stats in question are only ever incremented by one. >> That means that any calcluation related to overlap can be removed and >> special cased that it'll never overlap by more than 1. That potentially >> removes code that is required for other stats but not locality stats. >> This may give enough savings to avoid moving to s16. >> >> Very broadly speaking, I like what you're doing but I would like to see >> more work on reducing any unnecessary code in that path (such as dealing >> with overlaps for single increments) and treat incrasing the cache footprint >> only as a very last resort. >> Agree. I will think about it more. >>> #endif >>> #ifdef CONFIG_SMP >>> s8 stat_threshold; >>> diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h >>> index 1e19379..d97cc34 100644 >>> --- a/include/linux/vmstat.h >>> +++ b/include/linux/vmstat.h >>> @@ -125,10 +125,14 @@ static inline unsigned long global_numa_state(enum zone_numa_stat_item item) >>> return x; >>> } >>>