Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Nadav Amit <namit@vmware.com>
To:     kernel test robot <oliver.sang@intel.com>
CC:     Ingo Molnar <mingo@kernel.org>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        LKML <linux-kernel@vger.kernel.org>,
        "lkp@lists.01.org" <lkp@lists.01.org>,
        kernel test robot <lkp@intel.com>,
        "Huang, Ying" <ying.huang@intel.com>,
        "feng.tang@intel.com" <feng.tang@intel.com>,
        "zhengjun.xing@intel.com" <zhengjun.xing@intel.com>,
        Peter Zijlstra <peterz@infradead.org>
Subject: Re: [smp]  a32a4d8a81:  netperf.Throughput_tps -2.1% regression
Thread-Topic: [smp]  a32a4d8a81:  netperf.Throughput_tps -2.1% regression
Thread-Index: AQHXTLjYaZT7B2vV3EiePfaz1IztJKrrHQgA
Date:   Wed, 19 May 2021 18:17:35 +0000
Message-ID: <16A4D949-D82D-4034-B5A4-DE4DADE15884@vmware.com>
References: <20210519142742.GA5275@xsang-OptiPlex-9020>
In-Reply-To: <20210519142742.GA5275@xsang-OptiPlex-9020>
Accept-Language: en-US
Content-Language: en-US
Content-Type: multipart/signed;
        boundary="Apple-Mail=_58AAFC2A-66A2-47BB-84B3-9650423EA90C";
        protocol="application/pgp-signature";
        micalg=pgp-sha256
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 6e2a1454-f808-41e6-c948-08d91af260fc
X-MS-Exchange-CrossTenant-originalarrivaltime: 19 May 2021 18:17:36.0068
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: zzB68yFXeJv1D0HyVK/4MN6EEagK4faBp/7b6QzvkB3HKVNcGN51p5KPYDMqEKvdId1PhF0OS/kwIs4VpQpMTA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR05MB6248
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

--Apple-Mail=_58AAFC2A-66A2-47BB-84B3-9650423EA90C
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

[ +PeterZ for reference ]


> On May 19, 2021, at 7:27 AM, kernel test robot <oliver.sang@intel.com> =
wrote:
>=20
>=20
>=20
> Greeting,
>=20
> FYI, we noticed a -2.1% regression of netperf.Throughput_tps due to =
commit:
>=20
>=20
> commit: a32a4d8a815c4eb6dc64b8962dc13a9dfae70868 ("smp: Run functions =
concurrently in smp_call_function_many_cond()")
> =
https://nam04.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgit.ke=
rnel.org%2Fcgit%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git&amp;data=3D0=
4%7C01%7Cnamit%40vmware.com%7Ca49b22e928144aab039908d91acff8c4%7Cb39138ca3=
cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637570302823256266%7CUnknown%7CTWFpbGZsb=
3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C10=
00&amp;sdata=3Dh2VRetBNlEQBvOlkYrRCMCK6%2BukRqlCElYxM8UfVxqI%3D&amp;reserv=
ed=3D0 master
>=20
>=20
> in testcase: netperf
> on test machine: 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 =
CPU @ 2.30GHz with 192G memory
> with following parameters:
>=20
> 	ip: ipv4
> 	runtime: 300s
> 	nr_threads: 1
> 	cluster: cs-localhost
> 	test: UDP_RR
> 	cpufreq_governor: performance
> 	ucode: 0x5003006
>=20
>=20

[snip]

> commit:
>  v5.12-rc2
>  a32a4d8a81 ("smp: Run functions concurrently in =
smp_call_function_many_cond()")
>=20
>       v5.12-rc2 a32a4d8a815c4eb6dc64b8962dc
> ---------------- ---------------------------
>         %stddev     %change         %stddev
>             \          |                \
>    116903            -2.1%     114404        =
netperf.Throughput_total_tps
>    116903            -2.1%     114404        netperf.Throughput_tps
>  35066769            -2.1%   34317990        =
netperf.time.voluntary_context_switches
>  35071059            -2.1%   34321258        netperf.workload
>     67295            +1.5%      68333        proc-vmstat.nr_anon_pages
>    463520            -2.1%     453603        vmstat.system.cs
>    535.28 =C2=B1  6%      -8.3%     490.97 =C2=B1 10%  =
sched_debug.cfs_rq:/.util_est_enqueued.max
>      0.02 =C2=B1  8%     -10.8%       0.02 =C2=B1  4%  =
sched_debug.cpu.nr_running.avg
>  76309820 =C2=B1  4%    +320.0%  3.205e+08 =C2=B1158%  cpuidle.C1.time
>  23409116 =C2=B1  3%     +31.0%   30676822 =C2=B1 20%  =
cpuidle.C1.usage
>  46720133 =C2=B1  2%     -12.9%   40709940 =C2=B1  2%  =
cpuidle.POLL.usage
>      5282 =C2=B1110%    +317.0%      22029 =C2=B1 58%  =
numa-vmstat.node3.nr_anon_pages
>     11998 =C2=B1 55%    +138.7%      28637 =C2=B1 45%  =
numa-vmstat.node3.nr_inactive_anon
>     11998 =C2=B1 55%    +138.7%      28637 =C2=B1 45%  =
numa-vmstat.node3.nr_zone_inactive_anon
>      8397 =C2=B1136%    +588.7%      57827 =C2=B1 75%  =
numa-meminfo.node3.AnonHugePages
>     21162 =C2=B1110%    +316.7%      88189 =C2=B1 58%  =
numa-meminfo.node3.AnonPages
>     48780 =C2=B1 54%    +136.8%     115533 =C2=B1 45%  =
numa-meminfo.node3.Inactive
>     48780 =C2=B1 54%    +136.8%     115533 =C2=B1 45%  =
numa-meminfo.node3.Inactive(anon)
>    467040            -2.1%     457094        =
perf-stat.i.context-switches
>      0.01 =C2=B1138%      +0.0        0.03 =C2=B1 73%  =
perf-stat.i.dTLB-store-miss-rate%
> 9.415e+08            -2.4%  9.188e+08 =C2=B1  2%  =
perf-stat.i.dTLB-stores
>      0.01 =C2=B1137%      +0.0        0.03 =C2=B1 73%  =
perf-stat.overall.dTLB-store-miss-rate%
>    465472            -2.1%     455557        =
perf-stat.ps.context-switches
> 9.385e+08            -2.4%  9.158e+08 =C2=B1  2%  =
perf-stat.ps.dTLB-stores
>      1.21 =C2=B1 14%      +0.2        1.41 =C2=B1  5%  =
perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.=
sock_sendmsg.__sys_sendto
>      2.05 =C2=B1 10%      +0.3        2.33 =C2=B1  4%  =
perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.s=
tart_secondary.secondary_startup_64_no_verify
>      0.06 =C2=B1  7%      +0.0        0.08 =C2=B1 14%  =
perf-profile.children.cycles-pp.__calc_delta
>      0.08 =C2=B1 19%      +0.0        0.10 =C2=B1  9%  =
perf-profile.children.cycles-pp._copy_to_user
>      0.09 =C2=B1 22%      +0.0        0.12 =C2=B1  8%  =
perf-profile.children.cycles-pp._copy_from_user
>      0.12 =C2=B1 20%      +0.0        0.17 =C2=B1 13%  =
perf-profile.children.cycles-pp.exit_to_user_mode_prepare
>      0.14 =C2=B1 11%      +0.1        0.19 =C2=B1  9%  =
perf-profile.children.cycles-pp.skb_release_data
>      1.21 =C2=B1 14%      +0.2        1.41 =C2=B1  5%  =
perf-profile.children.cycles-pp.__ip_append_data
>      2.07 =C2=B1 11%      +0.3        2.33 =C2=B1  4%  =
perf-profile.children.cycles-pp.schedule_idle
>      0.06 =C2=B1  7%      +0.0        0.08 =C2=B1 11%  =
perf-profile.self.cycles-pp.__calc_delta
>      0.19 =C2=B1  8%      +0.0        0.24 =C2=B1  6%  =
perf-profile.self.cycles-pp.__softirqentry_text_start
>      0.24 =C2=B1  8%      +0.1        0.29 =C2=B1  4%  =
perf-profile.self.cycles-pp.__skb_recv_udp
>      0.14 =C2=B1 11%      +0.1        0.19 =C2=B1  9%  =
perf-profile.self.cycles-pp.skb_release_data
>      0.02 =C2=B1142%      +0.1        0.08 =C2=B1 17%  =
perf-profile.self.cycles-pp.sock_alloc_send_pskb
>      0.11 =C2=B1 17%      +0.1        0.19 =C2=B1 13%  =
perf-profile.self.cycles-pp.__ip_append_data
>      0.12 =C2=B1 34%      +0.1        0.26 =C2=B1 22%  =
perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
>      0.87 =C2=B1 13%      +0.2        1.05 =C2=B1  6%  =
perf-profile.self.cycles-pp._raw_spin_lock
>      1287 =C2=B1 42%     +75.3%       2256 =C2=B1 14%  =
interrupts.CPU111.CAL:Function_call_interrupts
>      1326 =C2=B1 43%     +71.0%       2267 =C2=B1 13%  =
interrupts.CPU119.CAL:Function_call_interrupts
>      1300 =C2=B1 45%     +75.9%       2287 =C2=B1 37%  =
interrupts.CPU120.CAL:Function_call_interrupts
>      1299 =C2=B1 45%     +60.1%       2081 =C2=B1 28%  =
interrupts.CPU128.CAL:Function_call_interrupts
>      1305 =C2=B1 45%     +61.7%       2110 =C2=B1 29%  =
interrupts.CPU131.CAL:Function_call_interrupts
>      1299 =C2=B1 45%     +61.8%       2102 =C2=B1 28%  =
interrupts.CPU139.CAL:Function_call_interrupts
>     66.67 =C2=B1133%     -97.2%       1.83 =C2=B1155%  =
interrupts.CPU14.TLB:TLB_shootdowns
>      1299 =C2=B1 45%    +107.8%       2700 =C2=B1 33%  =
interrupts.CPU142.CAL:Function_call_interrupts
>    301.83 =C2=B1128%     -95.6%      13.17 =C2=B1140%  =
interrupts.CPU149.RES:Rescheduling_interrupts
>    389.17 =C2=B1 89%     -73.5%     103.17 =C2=B1 35%  =
interrupts.CPU164.NMI:Non-maskable_interrupts
>    389.17 =C2=B1 89%     -73.5%     103.17 =C2=B1 35%  =
interrupts.CPU164.PMI:Performance_monitoring_interrupts
>      1299 =C2=B1 45%     +60.2%       2081 =C2=B1 28%  =
interrupts.CPU35.CAL:Function_call_interrupts
>      1244 =C2=B1 50%     +66.8%       2076 =C2=B1 27%  =
interrupts.CPU45.CAL:Function_call_interrupts
>      1300 =C2=B1 44%     +59.5%       2075 =C2=B1 28%  =
interrupts.CPU46.CAL:Function_call_interrupts
>      1.50 =C2=B1 63%   +1422.2%      22.83 =C2=B1167%  =
interrupts.CPU47.RES:Rescheduling_interrupts
>    467.33 =C2=B1 85%     -64.6%     165.67 =C2=B1 74%  =
interrupts.CPU58.NMI:Non-maskable_interrupts
>    467.33 =C2=B1 85%     -64.6%     165.67 =C2=B1 74%  =
interrupts.CPU58.PMI:Performance_monitoring_interrupts
>    306.67 =C2=B1 75%     -59.9%     122.83 =C2=B1 16%  =
interrupts.CPU68.NMI:Non-maskable_interrupts
>    306.67 =C2=B1 75%     -59.9%     122.83 =C2=B1 16%  =
interrupts.CPU68.PMI:Performance_monitoring_interrupts
>      1131 =C2=B1 27%     +61.2%       1822 =C2=B1 35%  =
interrupts.CPU85.CAL:Function_call_interrupts
>      1180 =C2=B1 31%     +79.6%       2119 =C2=B1 24%  =
interrupts.CPU86.CAL:Function_call_interrupts
>=20

Could it be a result of a regression that was resolved by commit
641acbf6fd6 ("smp: Micro-optimize smp_call_function_many_cond()")
or does this report mean that the performance regression also
happened on the -rc?


--Apple-Mail=_58AAFC2A-66A2-47BB-84B3-9650423EA90C
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEESJL3osl5Ymx/w9I1HaAqSabaD1oFAmClVj8ACgkQHaAqSaba
D1oyCA/9HxpRK1Li7xEVjsCcYv1fT9uq91N/EJhX96voDP+zIluULBrjCzqsfleT
8f51ISPiBy/TKwUHelpgswu22JWQFrBmryvxHdC+E+9KOcmysKtMXdGHQMls6vBF
Sj7ipW8gPhqy5wOi404SYztSYOChXtAQPEZiHcSXdD0JHlKNVEFY/czbfGNJYLC/
D0Q5JWXZWbleocplXFqRRHnvKhxUnFIMhCGqCQ5soNxWuT6LaalrncbKGizlG8V8
/qoHlySa02T51VYrwWTKASrcZ+GPsCxVNODl1Qf0XlAAjYiPjUwX43KLwRIpl4tZ
Jwrb0fVU8twVM1wWwV1hM5lod/3bIshXgLIn9NVu382+7PjO6biaVF0LJ80d91NO
0pFq4TPF4yg5sdbxzCRWf1DQ6BJ63+2ENipEzMp1S56SnOTFOusspDsrkGfFkvR3
5R1NHV6maAhkAkg4WbP8ogbberiK3ZWyXgcEdTEHUYhmMjw/Qyh5Qp58V7l2BtwM
sf6+P/caeWwx9Fs+qema2rMGr0m4LKlUNqeU0uLd/WpHsAn4AH9EYj0WEXmnPg4F
yYKPT6NCJ09CVCGdQD/hUmhiEXnCELtPRjs2NFWZxTOO4TQPkUzNg026tGbUCT8h
ZWn225WS9XYMCV/hH8FL5gaeQ0e2Zque5U2qUdmBi69BCmERUQk=
=iHLY
-----END PGP SIGNATURE-----

--Apple-Mail=_58AAFC2A-66A2-47BB-84B3-9650423EA90C--