Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <43598597-72e1-6750-9bee-031fd463fc3b@intel.com>
Date:   Wed, 27 Apr 2022 10:52:59 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.8.1
Subject: Re: [memcg] a8c49af3be: hackbench.throughput -13.7% regression
Content-Language: en-US
To:     kernel test robot <oliver.sang@intel.com>,
        Yosry Ahmed <yosryahmed@google.com>
CC:     Linus Torvalds <torvalds@linux-foundation.org>,
        Shakeel Butt <shakeelb@google.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Michal Hocko <mhocko@kernel.org>,
        Muchun Song <songmuchun@bytedance.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        LKML <linux-kernel@vger.kernel.org>, <lkp@lists.01.org>,
        <lkp@intel.com>, <ying.huang@intel.com>, <feng.tang@intel.com>,
        <zhengjun.xing@linux.intel.com>
References: <20220420055807.GA16310@xsang-OptiPlex-9020>
From:   Yin Fengwei <fengwei.yin@intel.com>
In-Reply-To: <20220420055807.GA16310@xsang-OptiPlex-9020>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MmdLVG1XOGRFdWV3bitJKzhMWkw3eHhXb1gyUWM3Z3VFZHJtVG1qWTl2aVdU?=
 =?utf-8?B?RlJsNnJUcjhqZ1Ayck9PUVJQR0gxQVhMa1FOZ0RVVUhTVDFNanZ3dUpnV1U3?=
 =?utf-8?B?endQQnRHQVUxMVhSZzZSMTZQSUN6SVRuellvcS9VOWQwWUd5bFFUMXh6Qks2?=
 =?utf-8?B?eW9rc2xKbUNxNzBVaXVhL1IvQytMc3dkaHhGMzhjWS9TeXRKeld1d2VXSXlM?=
 =?utf-8?B?MnhxQjhtOTB5STE5aUluSEtCYzJoRS8vM3AwRVphZlNmUGdFYlRpWXFrSXZX?=
 =?utf-8?B?MHdIZjFnN29uWE1EcWFKNm0relZ6MFg2b0NXSU0ySWJ3TndUeGprUDRiMlBB?=
 =?utf-8?B?T3FyaFBnZVplUWhkMDlia1E5RkFVSWVkSE5hdUdkM1QwQ3I0TS92RjNuK0VQ?=
 =?utf-8?B?T2R1T0dxeXB1Z2YrekRrSUtBZzZjTzdIaTZDRGZ6U2c0U2VLdGpqZkpCT2tP?=
 =?utf-8?B?eDZXMStTZzZyYngyUE1CUnpTSVVSdGxhRnROOUUxemZZbTZnVlBhcnBlOEhz?=
 =?utf-8?B?K0ZRQzNBNEZVaWR3dzdQZUsxU1JPOFU0bXdVNmFyZnF2NXdPcE9PT2NrcWxj?=
 =?utf-8?B?d0xmT0VNUnhWSzA1aTBtQjBNQkRHNkV6WjZNVDc0cWVTYVBPNEFBZDhqNlJx?=
 =?utf-8?B?TWtPRmNtb2JnZDB0Z2Fjdit5VnFJNndOSy9OckVYVjdsV2laYUFKa21pMzZi?=
 =?utf-8?B?VTZNakZuL25LNExLRDMxYUdLSFN3TlZ5VE5tT0dwMzN1SlMwT2w4TFpOZTkr?=
 =?utf-8?B?YjQ5M2p4UmFIV2FJS1lmaXRrWjlXa2g5ZmJUaTlWb1FrRlE4TlNBVXJXK0hn?=
 =?utf-8?B?OVB0bFkyb0RYcTRGQ05pSXErd2Q2emtEUnM5bFVvVTE4eHVrVXFkWTNNOG9t?=
 =?utf-8?B?TU1rdVRCMFJEQUttYXFZRENUbStYdmhrcTJTK1VnK1ZNT1VFQmJRazNxUnVD?=
 =?utf-8?B?ZGcxekJoQVlncjJITlk3MHh3STY2SVRTM0NOSGdBditqSGMwYmYxSHZLRVZs?=
 =?utf-8?B?N0JiNE43NXprc0IreVhCR1RJMzBXODBDZ1ZPZHlDUzkxQ3dES0psLzlvWXlq?=
 =?utf-8?B?d3lXVnZja3hsZmxxUCtrYXFseU5ZYW44T1dFRnRoNFJKdDNmek52cW5JQVJN?=
 =?utf-8?B?b3RpdnRQem9rYVZzS2V3UGtLVkVEUkhNcnUycnZ6TFp4NTlLWVNIczRLbjZo?=
 =?utf-8?B?aGllYVU1cjRGK0N3d1didE90Uk9TbWtXUDNiYWNNM0twS0JkOCtGUmF3cFNn?=
 =?utf-8?B?N2V0RFhhQ1MwNkljOC9xb3Vmc2tiQWp2K3pWRXFkSVY2WnNKZ2FvdDFNSjB4?=
 =?utf-8?B?bHFzUTZnL3dObldtTVErZmp4UHZuVGxuVUQ4Z3o1d2lZMkExbndCYlUyOGRo?=
 =?utf-8?B?VWcxdXZia1drMlJteFd1TmQ1eEt6WlNtM01rdVd4ZmpxSkNwei9PU21uU1pm?=
 =?utf-8?B?VkJQMlExYnNRanJsTjBQS2wyQWNIeEU1a0NyR2pyTk9YbTNWejMrMDFkZk13?=
 =?utf-8?B?Slhxa0Rxb21WZlRNWjRQQXE1MEhCUWNDOVJRK1lUSVdPVlBIaEtJMlJrS0hL?=
 =?utf-8?B?MTU3V2l1YmxEYnNrOFR0VnJQYTNtbHpSNG45UjdRQ2xXZ3VLek9mMzFYVDJQ?=
 =?utf-8?B?N2wyUjBuV1Q4ZmUvaFhhR3orN1pleUwxWGc2U3NueGs5bU9IMTQxblNIUU5O?=
 =?utf-8?B?RjNrTVpUdkhJeGpDQnhSdkV5ZlNuSXZqc1NnaTNUbEhIbDlOUXBSclZTSTd0?=
 =?utf-8?B?M0xlMmdNR2duVjFWanJKeG1yWEQveXlKUEMwTk9rR09YRDYxZFdBcmZUVlFD?=
 =?utf-8?B?dmlGaDRGVUY5aVNqaU1mT01qaE1HbnRwdWsxZGZHbGxlVmcvYWdaem1lOURN?=
 =?utf-8?B?Vnp2OXgzTVBGMndmaExvcWRyS0hoVTVuTmEvOExnREFMUDBSWjdOT05XSDhn?=
 =?utf-8?B?anpHc1lqYnBSMXdmdXB4eHFWakhLUmNEK1Y0MVlZNTRUVWNJME9sSTFCRXR3?=
 =?utf-8?B?bFdkc3pMU29vRi9oV0dhVWtPdEwrOVBickVqUitUVHNxUmk3MkVwR3p6ZWJX?=
 =?utf-8?B?YUVsRndmNFdqOFI1cHAzdHAwWVk1YU5OZENuR0paaWI1RHVzYmlsT09Md24w?=
 =?utf-8?B?TExFQjU4dXhsRHdCS2pCdFRwTDduYWpBV0d4NUt5UjB3ek8xMUdremtOZGl3?=
 =?utf-8?B?SFNPRFJlQTJmcHJNaGpTRks5NU1jWU9BOUVYbGZvR1greHJhTFEydDFCT0NP?=
 =?utf-8?B?Ym5KcEJIRGRyVXgyUmFxMDlySks5WnNnUDZYOFlTWi8zV1Jpa2lXRXcwVi93?=
 =?utf-8?B?OW1oVHQ0OHQ0V0FtazhweWRCVnZGbWdRWXFuazFxYnZ4OHFFalhwcGdtdi85?=
 =?utf-8?Q?Z8CCKtQQoRj3zRa8=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f7015f1d-61d7-413d-5275-08da27f90eda
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR11MB4831.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Apr 2022 02:53:08.0897
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: HrDyzoN9FhYhShqMQxbs0wfUZoEV1cSRyt+u4mdtaSscfSdSD07c01N8JAwTyMJo6nN5SmqNPzA1Y7q7WP7cOw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB3701
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Yosry,

On 4/20/2022 1:58 PM, kernel test robot wrote:
> 
> 
> Greeting,
> 
> FYI, we noticed a -13.7% regression of hackbench.throughput due to commit:
> 
> 
> commit: a8c49af3be5f0b4e105ef678bcf14ef102c270be ("memcg: add per-memcg total kernel memory stat")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> 
> in testcase: hackbench
> on test machine: 144 threads 4 sockets Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz with 128G memory
> with following parameters:
> 
> 	nr_threads: 100%
> 	iterations: 4
> 	mode: process
> 	ipc: socket
> 	cpufreq_governor: performance
> 	ucode: 0x7002402
> 
> test-description: Hackbench is both a benchmark and a stress test for the Linux kernel scheduler.
> test-url: https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/sched/cfs-scheduler/hackbench.c
> 
> 
> 
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <oliver.sang@intel.com>
> 
> 
> Details are as below:
> -------------------------------------------------------------------------------------------------->
> 
> 
> To reproduce:
> 
>         git clone https://github.com/intel/lkp-tests.git
>         cd lkp-tests
>         sudo bin/lkp install job.yaml           # job file is attached in this email
>         bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
>         sudo bin/lkp run generated-yaml-file
> 
>         # if come across any failure that blocks the test,
>         # please remove ~/.lkp and /lkp dir to run from a clean state.
> 
> =========================================================================================
> compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
>   gcc-11/performance/socket/4/x86_64-rhel-8.3/process/100%/debian-10.4-x86_64-20200603.cgz/lkp-cpl-4sp1/hackbench/0x7002402
> 
> commit: 
>   086f694a75 ("memcg: replace in_interrupt() with !in_task()")
>   a8c49af3be ("memcg: add per-memcg total kernel memory stat")
> 
> 086f694a75e1a283 a8c49af3be5f0b4e105ef678bcf 
> ---------------- --------------------------- 
>          %stddev     %change         %stddev
>              \          |                \  
>     146519           -13.7%     126397        hackbench.throughput
>     465.89           +16.0%     540.43        hackbench.time.elapsed_time
>     465.89           +16.0%     540.43        hackbench.time.elapsed_time.max
>  1.365e+08          +134.1%  3.195e+08 ±  4%  hackbench.time.involuntary_context_switches
>    1081515            -1.2%    1068489        hackbench.time.minor_page_faults
>      64911           +16.3%      75465        hackbench.time.system_time

Just FYI.

If I comment out one line added by the commit <a8c49af3be> :
static void memcg_account_kmem(struct mem_cgroup *memcg, int nr_pages)
{
        /* mod_memcg_state(memcg, MEMCG_KMEM, nr_pages); */  <--- comment out this line.
        if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) {
                if (nr_pages > 0)
                        page_counter_charge(&memcg->kmem, nr_pages);
                else
                        page_counter_uncharge(&memcg->kmem, -nr_pages);
        }
}

The regression is almost gone:

086f694a75e1a283 9ff9ec89a6dcf39f901ff0a84fd
---------------- ---------------------------
       fail:runs  %reproduction    fail:runs
           |             |             |
       7632:13      -44932%        1791:3     dmesg.timestamp:last
          1:13          -8%            :3     kmsg.common_interrupt:#No_irq_handler_for_vector
          2:13         -20%            :3     kmsg.timestamp:common_interrupt:#No_irq_handler_for_vector
       4072:13      -24827%         844:3     kmsg.timestamp:last
         %stddev     %change         %stddev
             \          |                \
    144327 ±  3%      -1.9%     141594 ±  5%  hackbench.throughput
                      regression dropped to -1.9% from -13.7%

    473.44 ±  3%      +1.9%     482.23 ±  5%  hackbench.time.elapsed_time


Regards
Yin, Fengwei