Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Song Liu <songliubraving@fb.com>
To:     Namhyung Kim <namhyung@kernel.org>
CC:     Arnaldo Carvalho de Melo <acme@kernel.org>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        Kernel Team <Kernel-team@fb.com>,
        "Arnaldo Carvalho de Melo" <acme@redhat.com>,
        Jiri Olsa <jolsa@kernel.org>
Subject: Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF
Thread-Topic: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF
Thread-Index: AQHXGqn3m2pRtlCATE6KZ0qUM3kt26qHp6sAgACBEACAAPZFAIAACxUAgAAqPoA=
Date:   Thu, 18 Mar 2021 07:03:41 +0000
Message-ID: <B8D1A735-BCE9-443A-8B38-31B0C0A6AA88@fb.com>
References: <20210316211837.910506-1-songliubraving@fb.com>
 <CAM9d7ci=hfFjq3+XuBcCZ0TUJxv6AmdFk0dkHFQD3wx27aJMjA@mail.gmail.com>
 <YFH//FRPvrPswhld@kernel.org> <7D48A756-C253-48DE-B536-826314778404@fb.com>
 <CAM9d7cjAngAKo9EazV=iyNncBZY53-rnE5_8SYuJiEuG4f4-yg@mail.gmail.com>
In-Reply-To: <CAM9d7cjAngAKo9EazV=iyNncBZY53-rnE5_8SYuJiEuG4f4-yg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <7C8F69ED0AD0614F92F5C7BB939260C8@namprd15.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: a7a8705c-3acd-43b6-aa02-08d8e9dbf690
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Mar 2021 07:03:41.5401
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: YOVc3rwMbwUdbr3yBpMfsj4N37vRNLiBw5X8kyBFToQq9y1bTI42A/FCpwyUM7V3+xb7wlRZEdZBI0i9gJaJBw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR15MB4203
X-OriginatorOrg: fb.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369,18.0.761
 definitions=2021-03-18_02:2021-03-17,2021-03-18 signatures=0
X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 phishscore=0
 mlxlogscore=999 suspectscore=0 malwarescore=0 impostorscore=0 mlxscore=0
 spamscore=0 clxscore=1015 bulkscore=0 adultscore=0 priorityscore=1501
 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2009150000 definitions=main-2103180052
X-FB-Internal: deliver
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> On Mar 17, 2021, at 9:32 PM, Namhyung Kim <namhyung@kernel.org> wrote:
>=20
> On Thu, Mar 18, 2021 at 12:52 PM Song Liu <songliubraving@fb.com> wrote:
>>=20
>>=20
>>=20
>>> On Mar 17, 2021, at 6:11 AM, Arnaldo Carvalho de Melo <acme@kernel.org>=
 wrote:
>>>=20
>>> Em Wed, Mar 17, 2021 at 02:29:28PM +0900, Namhyung Kim escreveu:
>>>> Hi Song,
>>>>=20
>>>> On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com> wrote=
:
>>>>>=20
>>>>> perf uses performance monitoring counters (PMCs) to monitor system
>>>>> performance. The PMCs are limited hardware resources. For example,
>>>>> Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.
>>>>>=20
>>>>> Modern data center systems use these PMCs in many different ways:
>>>>> system level monitoring, (maybe nested) container level monitoring, p=
er
>>>>> process monitoring, profiling (in sample mode), etc. In some cases,
>>>>> there are more active perf_events than available hardware PMCs. To al=
low
>>>>> all perf_events to have a chance to run, it is necessary to do expens=
ive
>>>>> time multiplexing of events.
>>>>>=20
>>>>> On the other hand, many monitoring tools count the common metrics (cy=
cles,
>>>>> instructions). It is a waste to have multiple tools create multiple
>>>>> perf_events of "cycles" and occupy multiple PMCs.
>>>>=20
>>>> Right, it'd be really helpful when the PMCs are frequently or mostly s=
hared.
>>>> But it'd also increase the overhead for uncontended cases as BPF progr=
ams
>>>> need to run on every context switch.  Depending on the workload, it ma=
y
>>>> cause a non-negligible performance impact.  So users should be aware o=
f it.
>>>=20
>>> Would be interesting to, humm, measure both cases to have a firm number
>>> of the impact, how many instructions are added when sharing using
>>> --bpf-counters?
>>>=20
>>> I.e. compare the "expensive time multiplexing of events" with its
>>> avoidance by using --bpf-counters.
>>>=20
>>> Song, have you perfmormed such measurements?
>>=20
>> I have got some measurements with perf-bench-sched-messaging:
>>=20
>> The system: x86_64 with 23 cores (46 HT)
>>=20
>> The perf-stat command:
>> perf stat -e cycles,cycles,instructions,instructions,ref-cycles,ref-cycl=
es <target, etc.>
>>=20
>> The benchmark command and output:
>> ./perf bench sched messaging -g 40 -l 50000 -t
>> # Running 'sched/messaging' benchmark:
>> # 20 sender and receiver threads per group
>> # 40 groups =3D=3D 1600 threads run
>>     Total time: 10X.XXX [sec]
>>=20
>>=20
>> I use the "Total time" as measurement, so smaller number is better.
>>=20
>> For each condition, I run the command 5 times, and took the median of
>> "Total time".
>>=20
>> Baseline (no perf-stat)                 104.873 [sec]
>> # global
>> perf stat -a                            107.887 [sec]
>> perf stat -a --bpf-counters             106.071 [sec]
>> # per task
>> perf stat                               106.314 [sec]
>> perf stat --bpf-counters                105.965 [sec]
>> # per cpu
>> perf stat -C 1,3,5                      107.063 [sec]
>> perf stat -C 1,3,5 --bpf-counters       106.406 [sec]
>>=20
>> From the data, --bpf-counters is slightly better than the regular event
>> for all targets. I noticed that the results are not very stable. There
>> are a couple 108.xx runs in some of the conditions (w/ and w/o
>> --bpf-counters).
>=20
> Hmm.. so this result is when multiplexing happened, right?
> I wondered how/why the regular perf stat is slower..

I should have made this more clear. This is when regular perf-stat time=20
multiplexing (2x ref-cycles on Intel). OTOH, bpf-counters does enables=20
sharing, so there is no time multiplexing. IOW, this is overhead of BPF=20
vs. overhead of time multiplexing.=20

Thanks,
Song=