Date:   Tue, 31 Jan 2023 14:11:58 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     Matthew Wilcox <willy@infradead.org>
CC:     "Sang, Oliver" <oliver.sang@intel.com>,
        Shakeel Butt <shakeelb@google.com>,
        "oe-lkp@lists.linux.dev" <oe-lkp@lists.linux.dev>,
        lkp <lkp@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Marek Szyprowski <m.szyprowski@samsung.com>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "linux-trace-kernel@vger.kernel.org" 
        <linux-trace-kernel@vger.kernel.org>,
        "Huang, Ying" <ying.huang@intel.com>,
        "zhengjun.xing@linux.intel.com" <zhengjun.xing@linux.intel.com>,
        "Yin, Fengwei" <fengwei.yin@intel.com>
Subject: Re: [linus:master] [mm]  f1a7941243:  unixbench.score -19.2%
 regression
Message-ID: <Y9ixLimDNWk9Rf6o@feng-clx>
References: <202301301057.e55dad5b-oliver.sang@intel.com>
 <Y9dETROtv9Bld9TI@casper.infradead.org>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <Y9dETROtv9Bld9TI@casper.infradead.org>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Q3lmaGdBeFNOOCtLNXZLRTUwa09YL3JLN1VNRUlTQ3lKM2tyNXRtbnljMlU4?=
 =?utf-8?B?MWh6SG5pbjZUWFdHSDVoVVZuN1p1N1dKb0EyK016SHE4VVNvRzVpUk0wbzIy?=
 =?utf-8?B?RkpLcUxzRXB3RVlDNWphVzJkdk15b25PcVg4MmpEbzJUZDBFRCsyUkltYkFi?=
 =?utf-8?B?RFQ0OVFCTWZNN3NWR1ZHUHJyenJrWWphd3VxSG5maWVDdlF0b2tZNC9rcUx2?=
 =?utf-8?B?cE1LVHZIWUFSazFvcUQ1enNOKzhBeTN2QjEyVms3VjVtK3A2RDU3SmFaUW9k?=
 =?utf-8?B?VEliQnZFSHo2U0FYWTVOODlUWFh0Wm93bXhYeUVPaEUzZW8rZU56L1RSZTJY?=
 =?utf-8?B?MlNudktNR1dTVXVvL3UxdFZDOFc5TWtCendWcVVnVHl4L1BWc2crcXRGaUJO?=
 =?utf-8?B?ZlhNRHlaNkdVWVFSUlpCeFhVZHcyUjJuWVVjSVpTeG42M2d3bkV2dDBOUUdy?=
 =?utf-8?B?QTFoQWNqOFFGRFFqM2ROaXlaVFZOcFA4YnQ4WlpscFRxNU1jSWVHS1UwK1ls?=
 =?utf-8?B?SC96UGpsM1E5TUkyTUVNQ1FEeFlQMWowYkdNZlhLK2ZmbzRzOFFhM3ArQUVs?=
 =?utf-8?B?b0dpNVlEL2dHc1E0aHlwa1J1eEYvOHZLbUgwZnFrTmx6YWYrZmhmWmprWThx?=
 =?utf-8?B?K2VqeXRUR2FtR1dKN3pxamFRdUtUU21oYnZYcHVrakZUYllMaG1COUhvcjlE?=
 =?utf-8?B?WHU4TS81VG43SWxiL2lUQlNERlFPM0xZVDNrVEZuanNGcUd6MmpScFlkb0Ix?=
 =?utf-8?B?Rmc0SXovR1FlaUNlZm5VblV2ZnVvUDVrZzFyenNvU1ZjcjVIR21MRjh1cmJI?=
 =?utf-8?B?SnUzOGNPRjhPc3QwY0JMWjZKMXI2aU9wTTBucE9nV1ROcGpaOWlUWmw3bnJx?=
 =?utf-8?B?MWlGSlNPbi82V2Z6ZnNTak9SMXNBOFRzcjFla3M5NVlmMzdmbzVpZEx6d29t?=
 =?utf-8?B?ZkhxVEJYSHpjdXhwSkJyVk5jTVdQYVlOTm9PRzhaVDB6UVJRM2RPS01PSHlP?=
 =?utf-8?B?dFRTTkR4QnNwNC9VT3ZUQncvQ0hyYWVCaHNaMkw1U29JenlrTml0a1RqZFY3?=
 =?utf-8?B?bGRValhvY2NsQmwra0pydXVGTmRHR25tbjhnQU5HcXU1dm5hdmpUQ2ZiWE1m?=
 =?utf-8?B?R21iMHNTQU1jVHFSTzlHL3Zkd2FlRHg2WjNiL0o0TUF4dktBbVlKZm9oUUFk?=
 =?utf-8?B?QldRaU9RMVppSndGaHJEaXBTZVNMOVJrQmRsUzVCeks3ZUsrcjJmYmFyRndP?=
 =?utf-8?B?WHUxM2FIKzZsQWRtb0NUZkdJcEZRYWlzS1NJMEV1aDl1ajFjTDFqNU80bmFh?=
 =?utf-8?B?Y0tsb2NjRmMrZHM4U2RTSnEzOW9OS2hXa0dDM2xvWXpYN2JTTDRXQlY2aTNB?=
 =?utf-8?B?OHJhdnV1eDZIRktVUlBmWDRhcy9GMzZpRkEyR2tlWURqQUl6ZHlXdjhPbXhF?=
 =?utf-8?B?dzlkcTNtd0VOUEFZa1p0SHZJSWtoVlhzSVQ5N0FaMDNXNjdVcGkyQnZRM3JI?=
 =?utf-8?B?dDVXMDVZdGJLQWIxNHcySWRKQ1V1eURLYy9wRUFjS2lUWEM0dVM1SXR6VVFi?=
 =?utf-8?B?blNKa3VXZzYzWVhwNFQ3TDVxMGVVNzhzL3FwY3d2SjNyOVZ0b0xwQUdBK2x6?=
 =?utf-8?B?dlUva040dXB5QysreDN3VThwZHhZaHB3MHdHQTRDblNuNVdrcVRSdjJrbmpD?=
 =?utf-8?B?NnFZSDMzNE5XckF2S2lnNXdQSk00RGJtempyNThIWkxVSHFVa1BqUFBiUi9U?=
 =?utf-8?B?dmpMWEVCanpFSG9FKzJ5N1NtZDZ1VGdxcmxVQ3dhb2dMQXlVRkJZYlk1ZUlJ?=
 =?utf-8?B?UWVuSmhkdDBXbjh6ZUFoTXJRUXR0Vk1OUEU5MnE4Tnh6VnIrMG5JT1FvOVdP?=
 =?utf-8?B?UzZtQW1IWnBjMHVCWjJBV3B3aHMrQ3ZyaEZKaVFYd3ZpNktlSzVJdERDU05l?=
 =?utf-8?B?ZUZ6SlEyVG14ci9PbklkMW9lV2ZEN0lHNmdCSDlYRXpXMmxQUVBrcWtBRnJS?=
 =?utf-8?B?ZUNoZlVjN1crclJWaWdYeW9hZDVMWXFGS1poS2o1Mk0yTU45V1poUjFVUkFO?=
 =?utf-8?B?aVduMkJaK1JzcCthZ1cyTnpZbVpLMjkwUXBlSlFWbFNTU2d6MnhxeEM5ZHBt?=
 =?utf-8?Q?iSil5S+FRPvK+ojM291XKtluu?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 9dcf071e-3993-4308-4185-08db03526b68
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Jan 2023 06:14:32.8971
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: l3cFo85vYS+0SxmjSoZStd4EOtO/O4wsIX+vqwgBO0jLk3Fj5P9fBTqt3jkD7kPx+ljn01QH+Y3hnUFe8M5PyQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB4514
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Jan 30, 2023 at 12:15:09PM +0800, Matthew Wilcox wrote:
> On Mon, Jan 30, 2023 at 10:32:56AM +0800, kernel test robot wrote:
> > FYI, we noticed a -19.2% regression of unixbench.score due to commit:
> > 
> > commit: f1a7941243c102a44e8847e3b94ff4ff3ec56f25 ("mm: convert mm's rss stats into percpu_counter")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > 
> > in testcase: unixbench
> > on test machine: 128 threads 4 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory
> > with following parameters:
> > 
> > 	runtime: 300s
> > 	nr_task: 30%
> > 	test: spawn
> > 	cpufreq_governor: performance
> 
> ...
> 
> > 9cd6ffa60256e931 f1a7941243c102a44e8847e3b94 
> > ---------------- --------------------------- 
> >          %stddev     %change         %stddev
> >              \          |                \  
> >      11110           -19.2%       8974        unixbench.score
> >    1090843           -12.2%     957314        unixbench.time.involuntary_context_switches
> >    4243909 ±  6%     -32.4%    2867136 ±  5%  unixbench.time.major_page_faults
> >      10547           -12.6%       9216        unixbench.time.maximum_resident_set_size
> >  9.913e+08           -19.6%  7.969e+08        unixbench.time.minor_page_faults
> >       5638           +19.1%       6714        unixbench.time.system_time
> >       5502           -20.7%       4363        unixbench.time.user_time
> 
> So we're spending a lot more time in the kernel and correspondingly less
> time in userspace.
> 
> >   67991885           -16.9%   56507507        unixbench.time.voluntary_context_switches
> >   46198768           -19.1%   37355723        unixbench.workload
> >  1.365e+08           -12.5%  1.195e+08 ±  7%  cpuidle..usage
> >    1220612 ±  4%     -38.0%     757009 ± 28%  meminfo.Active
> >    1220354 ±  4%     -38.0%     756754 ± 28%  meminfo.Active(anon)
> >       0.50 ±  2%      -0.1        0.45 ±  4%  mpstat.cpu.all.soft%
> >       1.73            -0.2        1.52 ±  2%  mpstat.cpu.all.usr%
> >     532266           -18.4%     434559        vmstat.system.cs
> >     495826           -12.2%     435455 ±  8%  vmstat.system.in
> >   1.36e+08           -13.2%   1.18e+08 ±  9%  turbostat.C1
> >      68.80            +0.8       69.60        turbostat.C1%
> >  1.663e+08           -12.1%  1.462e+08 ±  8%  turbostat.IRQ
> >      15.54 ± 20%     -49.0%       7.93 ± 24%  sched_debug.cfs_rq:/.runnable_avg.min
> >      13.26 ± 19%     -46.6%       7.08 ± 29%  sched_debug.cfs_rq:/.util_avg.min
> >      48.96 ±  8%     +51.5%      74.20 ± 13%  sched_debug.cfs_rq:/.util_est_enqueued.avg
> >     138.00 ±  5%     +28.9%     177.87 ±  7%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
> >     228060 ±  3%     +13.3%     258413 ±  4%  sched_debug.cpu.avg_idle.stddev
> >     432533 ±  5%     -16.4%     361517 ±  4%  sched_debug.cpu.nr_switches.min
> >  2.665e+08           -18.9%  2.162e+08        numa-numastat.node0.local_node
> >  2.666e+08           -18.9%  2.163e+08        numa-numastat.node0.numa_hit
> >  2.746e+08           -20.9%  2.172e+08        numa-numastat.node1.local_node
> >  2.747e+08           -20.9%  2.172e+08        numa-numastat.node1.numa_hit
> >  2.602e+08           -17.4%  2.149e+08        numa-numastat.node2.local_node
> >  2.603e+08           -17.4%  2.149e+08        numa-numastat.node2.numa_hit
> >  2.423e+08           -15.0%   2.06e+08        numa-numastat.node3.local_node
> >  2.424e+08           -15.0%  2.061e+08        numa-numastat.node3.numa_hit
> 
> So we're going off-node a lot more for ... something.
> 
> >  2.666e+08           -18.9%  2.163e+08        numa-vmstat.node0.numa_hit
> >  2.665e+08           -18.9%  2.162e+08        numa-vmstat.node0.numa_local
> >  2.747e+08           -20.9%  2.172e+08        numa-vmstat.node1.numa_hit
> >  2.746e+08           -20.9%  2.172e+08        numa-vmstat.node1.numa_local
> >  2.603e+08           -17.4%  2.149e+08        numa-vmstat.node2.numa_hit
> >  2.602e+08           -17.4%  2.149e+08        numa-vmstat.node2.numa_local
> >  2.424e+08           -15.0%  2.061e+08        numa-vmstat.node3.numa_hit
> >  2.423e+08           -15.0%   2.06e+08        numa-vmstat.node3.numa_local
> >     304947 ±  4%     -38.0%     189144 ± 28%  proc-vmstat.nr_active_anon
> 
> Umm.  Are we running vmstat a lot during this test?  The commit says:

Thanks for the hint.

Right. IIUC, 0Day normally will run 'vmstat 1' and collet info, and
also read /proc/vmstat and /sys/devices/system/node/nodexx/numstat
per second.

>     At the
>     moment the readers are either procfs interface, oom_killer and memory
>     reclaim which I think are not performance critical and should be ok with
>     slow read.  However I think we can make that change in a separate patch.
> 
> This would explain the increased cross-NUMA references (we're going to
> the other nodes to collect the stats), and the general slowdown.  But I
> don't think it reflects a real workload; it's reflecting that the
> monitoring of this workload that we're doing is now more accurate and
> more expensive.

We tried to disable the 3 types of 'vmstat/numastat' monitors
mentioned above. And the regression is still there with
similar rate around 19%. 

Thanks,
Feng