Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31;
Date:   Mon, 23 Oct 2023 09:25:12 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     Yosry Ahmed <yosryahmed@google.com>
CC:     Shakeel Butt <shakeelb@google.com>,
        "Sang, Oliver" <oliver.sang@intel.com>,
        "oe-lkp@lists.linux.dev" <oe-lkp@lists.linux.dev>,
        lkp <lkp@intel.com>,
        "cgroups@vger.kernel.org" <cgroups@vger.kernel.org>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "Huang, Ying" <ying.huang@intel.com>,
        "Yin, Fengwei" <fengwei.yin@intel.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Johannes Weiner <hannes@cmpxchg.org>,
        "Michal Hocko" <mhocko@kernel.org>,
        Roman Gushchin <roman.gushchin@linux.dev>,
        Muchun Song <muchun.song@linux.dev>,
        Ivan Babrou <ivan@cloudflare.com>, Tejun Heo <tj@kernel.org>,
        Michal =?utf-8?Q?Koutn=C3=BD?= <mkoutny@suse.com>,
        Waiman Long <longman@redhat.com>,
        "kernel-team@cloudflare.com" <kernel-team@cloudflare.com>,
        Wei Xu <weixugc@google.com>, Greg Thelen <gthelen@google.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v2 3/5] mm: memcg: make stats flushing threshold per-memcg
Message-ID: <ZTXLeAAI1chMamkU@feng-clx>
References: <20231010032117.1577496-4-yosryahmed@google.com>
 <202310202303.c68e7639-oliver.sang@intel.com>
 <CALvZod5hKvjm3WVSOGc5PpR9eNHFkt=BDmcrBe5CeWgFzP7jgQ@mail.gmail.com>
 <CAJD7tkbjZri4ayBOT9rJ0yMAi__c-1SVmRh_5oXezr7U6dvALg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAJD7tkbjZri4ayBOT9rJ0yMAi__c-1SVmRh_5oXezr7U6dvALg@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Q0FLdjhqL2pIRG9RdzZEVnpYZ1cxY3A3T0xPUVBEcDlrS0xtU296Zmc1UDFU?=
 =?utf-8?B?SnZpN096T1NJazluVFFRTk82d3dueWtadUMwN29kTDFOblJXT00zVndYaUQr?=
 =?utf-8?B?SW9aZDBCeU90V2lUWFhnWm1GWExZazFsNEV4WEVpTkEvelNrQUdSSUJTNWdK?=
 =?utf-8?B?VTNqREgzNm1wY1N1Y0JwSnJUR0hsRE9QZVo3K1ZBN1k1REV3aW9qTnBsTktY?=
 =?utf-8?B?Q3R3VXVnQURhSHFJMDY2MW1qTy9tNnRFSzJjRkl0VXg5czZLc1NveUVOOUZi?=
 =?utf-8?B?RW5WQW9QdTNJYlIzbC9zMW1pN0FqR3A1VFZPU3NIcERqdlhiaXB3a3IybGtx?=
 =?utf-8?B?QUxpTmF5MUpCY3lWTHYvZm9lME1nc1FsL1E0ZnlDVHo5ZGVhZHRFZG93djlJ?=
 =?utf-8?B?SUdFNmRVWlhhSmw2Q2l1MHpSTTROU2NtMk82U0FaYldTMVJxYzFQZXNZbHNB?=
 =?utf-8?B?MzFLUTFEd0NIeTlBem1iVGFRcTlZeHhkVW5jbFlOUVlpcnRMUXlSSXZqdFZ6?=
 =?utf-8?B?ck5iUWEwYmxXcmhwNWdzbHAxRXZlTmk5cTNsd0NNSnE4UFc3OW9lMnFwQVdY?=
 =?utf-8?B?Tkt2V1MyUFRjRmM2VTlqUG5DRSs3VEM1NEhYUUtRdTZRQ2wza09HbzhkYzZS?=
 =?utf-8?B?djNnbHliVi9JQkFiemx4dU1MOTNWbE44MkkxczhqcU84a0Z5SHhjZE9QVE9m?=
 =?utf-8?B?UmV4T3gvdVpVVkFTR1k2WlIxdTJPbWJYaUJPbS8yMDRRZnRPaXludmVOU2g3?=
 =?utf-8?B?MFZWaHZ5YkJ1NWVjM0ZyZndCcmVReCs2OGhIVTUvTmVlN2s5bXgwQUFBSXR3?=
 =?utf-8?B?dVZOSTdhQ2xUZW8wOE4yN2laNktDeDVjOGVSRTRDdnBVYWMyUWtCYXlucGRO?=
 =?utf-8?B?OGFvT0g4QjZlbE1lUktTMEhQUUNnRE1DZ0R1QU5UQWRQNHdaTW4wMTNMOU01?=
 =?utf-8?B?Zk5KbTU5NHNwOUJ0Y3JBOHVCZTlTUzZqdkI1dXV3YWkwQlAzN2UrU1pLckMy?=
 =?utf-8?B?V2ZoUC9vSkhvZnlBTG1BRkJoeG8yY2ZTVFdXU2pQZFFDMjlRYWVzMFkvWWxC?=
 =?utf-8?B?YVg5Q2VsL0RxR1Z1ZjN2U0xRc1ZzUGpYZ0ZTOU83Rlk2RFJqRDNGVERYcVls?=
 =?utf-8?B?eitBMnVNNjZOVXk4SWl3aGw0dktWalVkcGw1ZXQrUnJvWW1sbWJaK3Z2YUNp?=
 =?utf-8?B?bFhiSjF3WTRqZDA1aXlzYUtSaXhQZWNLa0ZrQnNMMXFhWmtkQm9MdDczaVhs?=
 =?utf-8?B?OElnY2VEbmJVc0JPdjdFVXRVVk1HTXo5VkpVOFo1QkZVSGpZQjNBbzZLOVR6?=
 =?utf-8?B?TjB4cG82WVBwcWNhK0ZRU2duRHZkd0JjMnYzbEU4ODJxWkVTT3F2RFl4UnZX?=
 =?utf-8?B?UHppVkhjNDd6TFZHNk0xQ1RTSFZVM1NTSHQ4V1NRelJyY0tGbGJIVHBTYWlF?=
 =?utf-8?B?UlE1ZVl2SmdmTjNlQ0NjQVFwOFNSQTErUW5aUnN5REcrWDI3VzJvcUxXcnY1?=
 =?utf-8?B?eTFrNGdGbVlOb3ordUZZdzJIc1dORjdqV0N6eHJybmoxR3RkdXJXREErL0JZ?=
 =?utf-8?B?UjhBcml0TVlueWxTaGs2ejlSQjIwVXBrNS85SEh4UXpoNkg2NjJaaHlObkpV?=
 =?utf-8?B?OG5BdlVXTTJiQXpMeDlMT245VTkvampldUZYQ2RZcVZnUHFPQndyN3hSUEp6?=
 =?utf-8?B?anpNeGsvdlB5MnZZQWxCcDJYRnROQ2x4S0lQUUFOQmh4M095cFNxbzJ6WGc3?=
 =?utf-8?B?bkdhcUdxMzlkTHAxR1l3ZkNIRGtwU2JrenRVZnVaTEQ3WGdWRllwRnJyYlpG?=
 =?utf-8?B?cVNJVC8xU3ozZXRnMjVjcnVrSlVFZXlRWnJaL1pyYno5NS94aVUxNzlVZDdD?=
 =?utf-8?B?Y3ZJRXllMDBwbHc0WnZ1OEF5RWVQZzYvMm5JSllNeG5Fek9Kdkh2dE9ncytY?=
 =?utf-8?B?dzIzbEVaMndvRm5vUXdud1A2cFljWmY4NUVHQkpHR1Q3VUl2aG10aFNwVUY4?=
 =?utf-8?B?ZklxOWRkT1NZUjZFbHB3YnhGQi9QV205UHB2RzZZR3NvdUNrM0dESGFoOFBw?=
 =?utf-8?B?cWVzTjEydkcwUDRFeWVkUXRuRGpqcEt0b09HNVplRi9pVEVMOGcydDI2R0gv?=
 =?utf-8?Q?nQtfcTT50By9YwLygW9dag+bd?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c69c9c30-d005-45a5-9a28-08dbd36833a1
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Oct 2023 01:34:30.0919
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: tjfLaqO8ldqQcPm4CTEPRH+iHRZ52LLpAt5+h/IMfyWhCDB6jt8dJUaFLoM1Ig/4P77COtR+NTpE0RSLDruP7Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB6746
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Sun, 22 Oct 2023 18:34:54 -0700 (PDT)

On Sat, Oct 21, 2023 at 01:42:58AM +0800, Yosry Ahmed wrote:
> On Fri, Oct 20, 2023 at 10:23 AM Shakeel Butt <shakeelb@google.com> wrote:
> >
> > On Fri, Oct 20, 2023 at 9:18 AM kernel test robot <oliver.sang@intel.com> wrote:
> > >
> > >
> > >
> > > Hello,
> > >
> > > kernel test robot noticed a -25.8% regression of will-it-scale.per_thread_ops on:
> > >
> > >
> > > commit: 51d74c18a9c61e7ee33bc90b522dd7f6e5b80bb5 ("[PATCH v2 3/5] mm: memcg: make stats flushing threshold per-memcg")
> > > url: https://github.com/intel-lab-lkp/linux/commits/Yosry-Ahmed/mm-memcg-change-flush_next_time-to-flush_last_time/20231010-112257
> > > base: https://git.kernel.org/cgit/linux/kernel/git/akpm/mm.git mm-everything
> > > patch link: https://lore.kernel.org/all/20231010032117.1577496-4-yosryahmed@google.com/
> > > patch subject: [PATCH v2 3/5] mm: memcg: make stats flushing threshold per-memcg
> > >
> > > testcase: will-it-scale
> > > test machine: 104 threads 2 sockets (Skylake) with 192G memory
> > > parameters:
> > >
> > >         nr_task: 100%
> > >         mode: thread
> > >         test: fallocate1
> > >         cpufreq_governor: performance
> > >
> > >
> > > In addition to that, the commit also has significant impact on the following tests:
> > >
> > > +------------------+---------------------------------------------------------------+
> > > | testcase: change | will-it-scale: will-it-scale.per_thread_ops -30.0% regression |
> > > | test machine     | 104 threads 2 sockets (Skylake) with 192G memory              |
> > > | test parameters  | cpufreq_governor=performance                                  |
> > > |                  | mode=thread                                                   |
> > > |                  | nr_task=50%                                                   |
> > > |                  | test=fallocate1                                               |
> > > +------------------+---------------------------------------------------------------+
> > >
> >
> > Yosry, I don't think 25% to 30% regression can be ignored. Unless
> > there is a quick fix, IMO this series should be skipped for the
> > upcoming kernel open window.
> 
> I am currently looking into it. It's reasonable to skip the next merge
> window if a quick fix isn't found soon.
> 
> I am surprised by the size of the regression given the following:
>       1.12 ą  5%      +1.4        2.50 ą  2%
> perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
> 
> IIUC we are only spending 1% more time in __mod_memcg_lruvec_state().

Yes, this is kind of confusing. And we have seen similar cases before,
espcially for micro benchmark like will-it-scale, stressng, netperf
etc, the change to those functions in hot path was greatly amplified
in the final benchmark score.

In a netperf case, https://lore.kernel.org/lkml/20220619150456.GB34471@xsang-OptiPlex-9020/
the affected functions have around 10% change in perf's cpu-cycles,
and trigger 69% regression. IIRC, micro benchmarks are very sensitive
to those statistics update, like memcg's and vmstat.

Thanks,
Feng