Received-SPF: pass (google.com: domain of linux-kernel+bounces-34819-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Tue, 23 Jan 2024 15:21:43 +0800
From: Oliver Sang <oliver.sang@intel.com>
To: Yosry Ahmed <yosryahmed@google.com>
CC: <oe-lkp@lists.linux.dev>, <lkp@intel.com>, <linux-kernel@vger.kernel.org>,
	Andrew Morton <akpm@linux-foundation.org>, Johannes Weiner
	<hannes@cmpxchg.org>, Domenico Cerasuolo <cerasuolodomenico@gmail.com>,
	Shakeel Butt <shakeelb@google.com>, Chris Li <chrisl@kernel.org>, Greg Thelen
	<gthelen@google.com>, Ivan Babrou <ivan@cloudflare.com>, Michal Hocko
	<mhocko@kernel.org>, Michal Koutny <mkoutny@suse.com>, Muchun Song
	<muchun.song@linux.dev>, Roman Gushchin <roman.gushchin@linux.dev>, Tejun Heo
	<tj@kernel.org>, Waiman Long <longman@redhat.com>, Wei Xu
	<weixugc@google.com>, <cgroups@vger.kernel.org>, <linux-mm@kvack.org>,
	<ying.huang@intel.com>, <feng.tang@intel.com>, <fengwei.yin@intel.com>,
	<oliver.sang@intel.com>
Subject: Re: [linus:master] [mm] 8d59d2214c: vm-scalability.throughput -36.6%
 regression
Message-ID: <Za9pB928KjSORPw+@xsang-OptiPlex-9020>
References: <202401221624.cb53a8ca-oliver.sang@intel.com>
 <CAJD7tka0o+jn3UkXB+ZfZvRw1v+KysJbaGQvJdHcSmAhYC5TQA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAJD7tka0o+jn3UkXB+ZfZvRw1v+KysJbaGQvJdHcSmAhYC5TQA@mail.gmail.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Ky9PVFV3eUE4czJyM1orWnVaNk5pa200REpJczkvMm9ZSzJNQzE4OHlvM2Nz?=
 =?utf-8?B?VTZMVTBWUUN5S3hubVFxUnN6ZDkxczFMZ3ZKYS9PYjE5czFtM1lFdXlVeHFa?=
 =?utf-8?B?dUhrQ25KQkw0Zkw2aGs4WWl5R2tKMURWcHEyd0hJRDIxa3JkajlXOHdHd1Ur?=
 =?utf-8?B?eExmTm5YczhpZEhYRGUxWXZQVVdSeGZqaGZLZ3Z1Q2lwYWFWRTlxZHBqUk8z?=
 =?utf-8?B?R0tzU0cwSU82aVREV0FjTmlEYmdxMzVmV2d1WUFCWW9tNDYrVjUvYmEybFZJ?=
 =?utf-8?B?VWhoT1daNzN1YXY1Ujl4NTNSWkQ5d1lSeS9VWTVSQk9wRnQ1NXNnZ0lKUmUz?=
 =?utf-8?B?ZDRSL3U1aWRtaEcybjJ1STd2OVZjRmpBOS9MZThKQk44SjZ1QzNCVzVrU0tz?=
 =?utf-8?B?YmM5UXFDOFJlWksvR0NteEkvMTRKSy9vc1R1MEdOcVF4NnJ6QzY3U2VlNEtq?=
 =?utf-8?B?d2E5MVZ5R2g4TEdLYlozRUI3SlB1SWNXNUY0YW4ycmtnRER1eDB5clpnZTZi?=
 =?utf-8?B?aVhJTTVpK2FsZlQrY0JEWThUR1pjVm5rTU9EQkIxRjh1Z0g2dVlqUHprSWYx?=
 =?utf-8?B?NC95aDQwSXpBNUVUOUJMSlFNRWFmR2V3TnRrS2c2K1kzRmJUTEU3dU16V1k0?=
 =?utf-8?B?eWwrSUZ4SHBxVzRqT1lDNGRFOTJFejIzeHJ0WEZ3ZVJ2eTdvai9YVmtwV1A3?=
 =?utf-8?B?NzlYOE5hb0taT2UzdmFrQzNwWEZPS09UL241T3dQZE4zMnFsUFl1bHRiL2w1?=
 =?utf-8?B?elZTNjkxWFpWOW5ZcnNyNHMvUVBXYUxhcERpci9sckZhSUJWdlprd25QVDVZ?=
 =?utf-8?B?UlgxVmQyR21FS05QOUFPa1FLVEFrZEJyZ0dJTWdBc3BaR1h1NGo0OGNWSXdT?=
 =?utf-8?B?a3c1UE5sUVNOclZ1UGZVMnc5aFJLcnBxTTd2WUI5RDlEYUIyZHkyL2w1ZFZi?=
 =?utf-8?B?YzZhWjhpNTJGck41WjRJdnJPTXVQd0xOTDd1c1hFQjB0ajBhNlptUDBsazc3?=
 =?utf-8?B?a3FsUDJCL0RPYm16d0FnQkN1RHhKUmRkdlRIV2VreHNKUUhaUURPbVQvTFVy?=
 =?utf-8?B?ekQ1YXVQaVBORGp3QlN1N3pydUoxSHRWbDY5Q1JWNkVKR0ZIUER1ZmM2U29W?=
 =?utf-8?B?c2xCeWNOZzFnaVV5dGlGWDBNcVVXMFp4eFpIRStoaVZLMEU4Q2dSZStMTFJL?=
 =?utf-8?B?WnB1bDFzVFVGd0JRRlRBRXk1bjczLytZZjRJTytGMkpyVStJZ2pKSzRpQTNZ?=
 =?utf-8?B?YXJnNFQ5MjdBUExOT3pvS1MzQk52NTBnZS9BZzhVdER4VnBOTnZ2TWpSUk1P?=
 =?utf-8?B?MVBORmtpcURzcVpDRzFBbkpmUWhQRUc2dnYzSXA2dHlkaGVrTVp4Y0RLVndW?=
 =?utf-8?B?Z3ZDdTlNajF5TzdKSWEzcjFwdzdRVzR2U1I5bCtaOEFFYmY2aDM4VjJBaWpZ?=
 =?utf-8?B?ZjBFK1RJVEVNbHZjd0JwSlZIaXF4SFE0c1Ywa0hFL0hMbmhTYjY5dDh1OTBn?=
 =?utf-8?B?M0pvOURLcHVSU1c3NlVVaHFLL00rY3l3UkNhaVVuYTF6Z21XdDA3bjlwdFpE?=
 =?utf-8?B?a09KRDRvT3kzOS82L3dFak02c09KSU5VRmJpdjRPdUROQjZYd2c1SEU0Z2Yz?=
 =?utf-8?B?YnhZQVFaMzdMY1lrRVFUUHhIZUVSay9HRG8zbU1EOFArbnZHMFFBelF1Y1R4?=
 =?utf-8?B?TUFVMWZRK3ljRzdsOStCbnhsaUVQb2t3QUV1eTVjbCtxcml5UWFIa0JFUGtT?=
 =?utf-8?B?V2Z6bEhPZGIzdGtDTDFEbm4yeDhvUVVBNUJmV2VJQkdGUVJ1QXZRQW9leksr?=
 =?utf-8?B?aVpqdlFJMndnRTdSbDYvTlRieElMTmloclBTTTBDSVJRWjRvazc4VnNJTGlp?=
 =?utf-8?B?aGJKOEswRFFZWmxZR2FrZ0d4VDgrNjg2TzlaRFdxempmOFZFMzR1UUdLNlJX?=
 =?utf-8?B?UHFtTStKeittTi9NQVg0RWYzN3RXUW1GUDF5amNHTGdyczFGVHhlcTZuV0VG?=
 =?utf-8?B?WDNaSmpoTVF6eCtwT1F6N1VIamdWTC9WdUpLTmNoMm5tcXlVTHBVcmhpQjBX?=
 =?utf-8?B?RHUwK2d0N1Q2NTc0L1Q4em15RTNEZUIvUWVTNklmaU9QdjBUVk9FOTFneXhX?=
 =?utf-8?B?NDdneGJtWDJ0a3I0T0d4dHkzK2ppRWlSUWVaWGFDT0dUOHk1ditFY3JRTU5C?=
 =?utf-8?B?N1E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 067e2ff2-9c2c-4321-5e41-08dc1be3faf3
X-MS-Exchange-CrossTenant-AuthSource: LV3PR11MB8603.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jan 2024 07:21:56.2469
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: XMkT+PsHwffSrv9nsnHLhaCeSdTZLPuvp62/314GMvxvonRvef+AqJPh2l/RBSxRejYLDmiHtDylVsbsNV4YhQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB6397
X-OriginatorOrg: intel.com

hi, Yosry Ahmed,

On Mon, Jan 22, 2024 at 01:39:19PM -0800, Yosry Ahmed wrote:
> On Mon, Jan 22, 2024 at 12:39 AM kernel test robot
> <oliver.sang@intel.com> wrote:
> >
> >
> >
> > hi, Yosry Ahmed,
> >
> > per your suggestion in
> > https://lore.kernel.org/all/CAJD7tkameJBrJQxRj+ibKL6-yd-i0wyoyv2cgZdh3ZepA1p7wA@mail.gmail.com/
> > "I think it would be useful to know if there are
> > regressions/improvements in other microbenchmarks, at least to
> > investigate whether they represent real regressions."
> >
> > we still report below two regressions to you just FYI what we observed in our
> > microbenchmark tests.
> > (we still captured will-it-scale::fallocate regression but ignore here per
> > your commit message)
> >
> >
> > Hello,
> >
> > kernel test robot noticed a -36.6% regression of vm-scalability.throughput on:
> >
> >
> > commit: 8d59d2214c2362e7a9d185d80b613e632581af7b ("mm: memcg: make stats flushing threshold per-memcg")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> >
> > testcase: vm-scalability
> > test machine: 224 threads 4 sockets Intel(R) Xeon(R) Platinum 8380H CPU @ 2.90GHz (Cooper Lake) with 192G memory
> > parameters:
> >
> >         runtime: 300s
> >         size: 1T
> >         test: lru-shm
> >         cpufreq_governor: performance
> >
> > test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
> > test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/
> >
> > In addition to that, the commit also has significant impact on the following tests:
> >
> > +------------------+----------------------------------------------------------------------------------------------------+
> > | testcase: change | will-it-scale: will-it-scale.per_process_ops -32.3% regression                                     |
> > | test machine     | 104 threads 2 sockets (Skylake) with 192G memory                                                   |
> > | test parameters  | cpufreq_governor=performance                                                                       |
> > |                  | mode=process                                                                                       |
> > |                  | nr_task=50%                                                                                        |
> > |                  | test=tlb_flush2                                                                                    |
> > +------------------+----------------------------------------------------------------------------------------------------+
> >
> >
> > If you fix the issue in a separate patch/commit (i.e. not just a new version of
> > the same patch/commit), kindly add following tags
> > | Reported-by: kernel test robot <oliver.sang@intel.com>
> > | Closes: https://lore.kernel.org/oe-lkp/202401221624.cb53a8ca-oliver.sang@intel.com
> 
> Thanks for reporting this. We have had these patches running on O(10K)
> machines in our production for a while now, and there haven't been any
> complaints (at least not yet). OTOH, we do see significant CPU savings
> on reading memcg stats.
> 
> That being said, I think we can improve the performance here by
> caching pointers to the parent_memcg->vmstats_percpu and
> memcg->vmstats in struct memcg_vmstat_percpu. This should
> significantly reduce the memory fetches in the loop in
> memcg_rstat_updated().
> 
> Oliver, would you be able to test if the attached patch helps? It's
> based on 8d59d2214c236.

the patch failed to compile:

build_errors:
  - "mm/memcontrol.c:731:38: error: 'x' undeclared (first use in this function)"


> 
> [..]