Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Nadav Amit <namit@vmware.com>
To:     Peter Zijlstra <peterz@infradead.org>
CC:     kernel test robot <oliver.sang@intel.com>,
        Ingo Molnar <mingo@kernel.org>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        LKML <linux-kernel@vger.kernel.org>,
        "lkp@lists.01.org" <lkp@lists.01.org>,
        kernel test robot <lkp@intel.com>,
        "Huang, Ying" <ying.huang@intel.com>,
        "feng.tang@intel.com" <feng.tang@intel.com>,
        "zhengjun.xing@intel.com" <zhengjun.xing@intel.com>
Subject: Re: [smp]  a32a4d8a81:  netperf.Throughput_tps -2.1% regression
Thread-Topic: [smp]  a32a4d8a81:  netperf.Throughput_tps -2.1% regression
Thread-Index: AQHXTLjYaZT7B2vV3EiePfaz1IztJKrrHQgAgAAF5ICAAA6zgA==
Date:   Wed, 19 May 2021 19:31:16 +0000
Message-ID: <CDFF26F9-A748-403C-86B2-4511DD6B1833@vmware.com>
References: <20210519142742.GA5275@xsang-OptiPlex-9020>
 <16A4D949-D82D-4034-B5A4-DE4DADE15884@vmware.com>
 <20210519183839.GH21560@worktop.programming.kicks-ass.net>
In-Reply-To: <20210519183839.GH21560@worktop.programming.kicks-ass.net>
Accept-Language: en-US
Content-Language: en-US
Content-Type: multipart/signed;
        boundary="Apple-Mail=_47B24B75-12F4-455D-953C-9FC49F9C7AD0";
        protocol="application/pgp-signature";
        micalg=pgp-sha256
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 319d48c2-dd63-44c4-4b56-08d91afcab82
X-MS-Exchange-CrossTenant-originalarrivaltime: 19 May 2021 19:31:16.0263
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: cnmMQkS2vUtDch9SPESZ0U95Nvu866wUobKBTw7Vr1ebF02Nelt++hNeVas64OkEnnQ/5malN89kShxEg94Wdw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR05MB5478
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

--Apple-Mail=_47B24B75-12F4-455D-953C-9FC49F9C7AD0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8


> On May 19, 2021, at 11:38 AM, Peter Zijlstra <peterz@infradead.org> =
wrote:
>=20
> On Wed, May 19, 2021 at 06:17:35PM +0000, Nadav Amit wrote:
>>>     1287 =C2=B1 42%     +75.3%       2256 =C2=B1 14%  =
interrupts.CPU111.CAL:Function_call_interrupts
>>>     1326 =C2=B1 43%     +71.0%       2267 =C2=B1 13%  =
interrupts.CPU119.CAL:Function_call_interrupts
>>>     1300 =C2=B1 45%     +75.9%       2287 =C2=B1 37%  =
interrupts.CPU120.CAL:Function_call_interrupts
>>>     1299 =C2=B1 45%     +60.1%       2081 =C2=B1 28%  =
interrupts.CPU128.CAL:Function_call_interrupts
>>>     1305 =C2=B1 45%     +61.7%       2110 =C2=B1 29%  =
interrupts.CPU131.CAL:Function_call_interrupts
>>>     1299 =C2=B1 45%     +61.8%       2102 =C2=B1 28%  =
interrupts.CPU139.CAL:Function_call_interrupts
>>>    66.67 =C2=B1133%     -97.2%       1.83 =C2=B1155%  =
interrupts.CPU14.TLB:TLB_shootdowns
>>>     1299 =C2=B1 45%    +107.8%       2700 =C2=B1 33%  =
interrupts.CPU142.CAL:Function_call_interrupts
>>>   301.83 =C2=B1128%     -95.6%      13.17 =C2=B1140%  =
interrupts.CPU149.RES:Rescheduling_interrupts
>>>   389.17 =C2=B1 89%     -73.5%     103.17 =C2=B1 35%  =
interrupts.CPU164.NMI:Non-maskable_interrupts
>>>   389.17 =C2=B1 89%     -73.5%     103.17 =C2=B1 35%  =
interrupts.CPU164.PMI:Performance_monitoring_interrupts
>>>     1299 =C2=B1 45%     +60.2%       2081 =C2=B1 28%  =
interrupts.CPU35.CAL:Function_call_interrupts
>>>     1244 =C2=B1 50%     +66.8%       2076 =C2=B1 27%  =
interrupts.CPU45.CAL:Function_call_interrupts
>>>     1300 =C2=B1 44%     +59.5%       2075 =C2=B1 28%  =
interrupts.CPU46.CAL:Function_call_interrupts
>>>     1.50 =C2=B1 63%   +1422.2%      22.83 =C2=B1167%  =
interrupts.CPU47.RES:Rescheduling_interrupts
>>>   467.33 =C2=B1 85%     -64.6%     165.67 =C2=B1 74%  =
interrupts.CPU58.NMI:Non-maskable_interrupts
>>>   467.33 =C2=B1 85%     -64.6%     165.67 =C2=B1 74%  =
interrupts.CPU58.PMI:Performance_monitoring_interrupts
>>>   306.67 =C2=B1 75%     -59.9%     122.83 =C2=B1 16%  =
interrupts.CPU68.NMI:Non-maskable_interrupts
>>>   306.67 =C2=B1 75%     -59.9%     122.83 =C2=B1 16%  =
interrupts.CPU68.PMI:Performance_monitoring_interrupts
>>>     1131 =C2=B1 27%     +61.2%       1822 =C2=B1 35%  =
interrupts.CPU85.CAL:Function_call_interrupts
>>>     1180 =C2=B1 31%     +79.6%       2119 =C2=B1 24%  =
interrupts.CPU86.CAL:Function_call_interrupts
>>>=20
>=20
> It looks to be sending *waay* more call IPIs, did we mess up the mask =
or
> loose an optimization somewhere?
>=20
> I'll go read the commit again=E2=80=A6

As you know, I did mess up by calling =
arch_send_call_function_single_ipi()
instead of smp_call_function_single(), which could explain the extra =
IPIs.
But that was resolved by your subsequent patch.

For me, what stands out is the time in C1 spent after the patch.

I will try to reproduce the issue to figure it out, since so far I could
not find an error in the code.


--Apple-Mail=_47B24B75-12F4-455D-953C-9FC49F9C7AD0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEESJL3osl5Ymx/w9I1HaAqSabaD1oFAmClZ4MACgkQHaAqSaba
D1q+qQ/5ASUnZGF2Piy5f1+ULAa19WtBFWgf5jL434a1fh+z/6KLpXj4HFMLrCaN
+i1GKT8kCfGl7UcIJS3xhd8s0A1F5390BJ/SIESSM7Me64o67SzsUoh057iC66B/
YkeTArgmPGjz9MAARxxoXRNPdMwx9dgDft5NgS5U8X2Qe1c+RzLKLXGwnNQB6waL
H30nFmWBdTDVLxpy4FvODs8TIK5kywCW9a/kb6rq6KHDFynBnJVfR8IZIFpHMxQ4
nJmSMVZqcYwkLQZLFWMS8yxul/D3pGKyW5nGdJFNbSAnqikhCtUPatuR3NBCcPBI
VUzB1XqPcbPMG/X42PCpsWSDYg2i5msbd5f5+MbZ4b0S2z0Rv4F1ySsWd+LpW/Si
K7ET0MntsuwuftQ2itPNJKovnWu8ynXy89eiw1SYiuS7nmnqOrC23P+I0nyDktSg
QhUt2gJj0VgiFc9xkCRS06v0H2iO+DtF//YT4HszMFJxw8S4cqdA059IxtdkH2ny
RmR3xqqlcgggXXZh4rFPMoiJJe7oxXiF7zhYWERxWzQL4ak4UDCkHorjU02ax4GL
Oz0J0Mjg2AeIkFrtt+aBh/b5qb8lGRDA8OJAKxcaXr9F9yYnv3VO8zXG4RIWdNBm
bLTXT2wTRMFa8CKNeVsBA+CHWPMl6oH6tOS0qWGJZrkznCTDDh0=
=xXQB
-----END PGP SIGNATURE-----

--Apple-Mail=_47B24B75-12F4-455D-953C-9FC49F9C7AD0--