New architectures bring new complexity, such as Intel's hybrid
models. jevents provides an alternative to specifying events in the
kernel and exposing them through sysfs, however, it is difficult to
work with. For example, an error in the json input would yield an
error message but no json file or location. It is also a challenge to
update jsmn.c given its forked nature.
The changes here switch from jevents.c to a rewrite in python called
jevents.py. This means there is a build time dependency on python, but
such a dependency already exists for asciidoc (used to generate perf's
man pages). If the build detects that python isn't present or is older
than version 3.6 (released Dec. 2016) then an empty file is
substituted for the generated one.
A challenge with this code is in avoiding regressions. For this reason
the jevents.py produces identical output to jevents.c, validated with a
test script and build target.
v6. Fixes an annotation and use of removesuffix that aren't present in
Python 3.6. Linter issues are also fixed.The code was tested on
Python 3.6 and 3.8 with docker.
v5. Adds a 2>/dev/null as suggested by David Laight
<[email protected]>.
v4. Fixes the build for systems with python2 installed by adding a
patch that makes python3 the preferred python (Reported-by: John
Garry <[email protected]>). It also fixes a bash-ism in the
jevents-test.sh and fixes the handling of an empty string for a
metric BriefDescription as one was added for sapphirerapids in the
metric Execute.
v3. Updates the patches for merged changes (on
acme/tmp.perf/core). Re-runs all comparisons to make sure the
generated pmu-events.c isn't altered at all by this change. Adds
the jevents.c ExtSel fix in:
https://lore.kernel.org/lkml/[email protected]/
Bumps the python version from 3.5 to 3.6, as f-strings weren't
introduced until 3.6.
v2. Fixes the build for architectures that don't have pmu-events json
(Suggested-by: John Garry <[email protected]>) and fixes the
build for python not being present or too old (Suggested-by: Peter
Zijlstra <[email protected]>/John Garry <[email protected]>).
Ian Rogers (4):
perf python: Prefer python3
perf jevents: Add python converter script
perf jevents: Switch build to use jevents.py
perf jevents: Remove jevents.c
tools/perf/Makefile.config | 27 +-
tools/perf/Makefile.perf | 16 +-
tools/perf/pmu-events/Build | 15 +-
tools/perf/pmu-events/empty-pmu-events.c | 21 +
tools/perf/pmu-events/jevents.c | 1342 ----------------------
tools/perf/pmu-events/jevents.py | 409 +++++++
tools/perf/pmu-events/jsmn.h | 68 --
tools/perf/pmu-events/json.c | 162 ---
tools/perf/pmu-events/json.h | 39 -
9 files changed, 465 insertions(+), 1634 deletions(-)
create mode 100644 tools/perf/pmu-events/empty-pmu-events.c
delete mode 100644 tools/perf/pmu-events/jevents.c
create mode 100755 tools/perf/pmu-events/jevents.py
delete mode 100644 tools/perf/pmu-events/jsmn.h
delete mode 100644 tools/perf/pmu-events/json.c
delete mode 100644 tools/perf/pmu-events/json.h
--
2.36.1.476.g0c4daa206d-goog
Hi Ian,
On 6/18/2022 8:09 AM, Ian Rogers wrote:
> New architectures bring new complexity, such as Intel's hybrid
> models. jevents provides an alternative to specifying events in the
> kernel and exposing them through sysfs, however, it is difficult to
> work with. For example, an error in the json input would yield an
> error message but no json file or location. It is also a challenge to
> update jsmn.c given its forked nature.
>
> The changes here switch from jevents.c to a rewrite in python called
> jevents.py. This means there is a build time dependency on python, but
> such a dependency already exists for asciidoc (used to generate perf's
> man pages). If the build detects that python isn't present or is older
> than version 3.6 (released Dec. 2016) then an empty file is
> substituted for the generated one.
>
> A challenge with this code is in avoiding regressions. For this reason
> the jevents.py produces identical output to jevents.c, validated with a
> test script and build target.
>
> v6. Fixes an annotation and use of removesuffix that aren't present in
> Python 3.6. Linter issues are also fixed.The code was tested on
> Python 3.6 and 3.8 with docker.
I just test it, now it can be built successfully with python 3.8.
> v5. Adds a 2>/dev/null as suggested by David Laight
> <[email protected]>.
> v4. Fixes the build for systems with python2 installed by adding a
> patch that makes python3 the preferred python (Reported-by: John
> Garry <[email protected]>). It also fixes a bash-ism in the
> jevents-test.sh and fixes the handling of an empty string for a
> metric BriefDescription as one was added for sapphirerapids in the
> metric Execute.
> v3. Updates the patches for merged changes (on
> acme/tmp.perf/core). Re-runs all comparisons to make sure the
> generated pmu-events.c isn't altered at all by this change. Adds
> the jevents.c ExtSel fix in:
> https://lore.kernel.org/lkml/[email protected]/
> Bumps the python version from 3.5 to 3.6, as f-strings weren't
> introduced until 3.6.
>
> v2. Fixes the build for architectures that don't have pmu-events json
> (Suggested-by: John Garry <[email protected]>) and fixes the
> build for python not being present or too old (Suggested-by: Peter
> Zijlstra <[email protected]>/John Garry <[email protected]>).
>
> Ian Rogers (4):
> perf python: Prefer python3
> perf jevents: Add python converter script
> perf jevents: Switch build to use jevents.py
> perf jevents: Remove jevents.c
>
> tools/perf/Makefile.config | 27 +-
> tools/perf/Makefile.perf | 16 +-
> tools/perf/pmu-events/Build | 15 +-
> tools/perf/pmu-events/empty-pmu-events.c | 21 +
> tools/perf/pmu-events/jevents.c | 1342 ----------------------
> tools/perf/pmu-events/jevents.py | 409 +++++++
> tools/perf/pmu-events/jsmn.h | 68 --
> tools/perf/pmu-events/json.c | 162 ---
> tools/perf/pmu-events/json.h | 39 -
> 9 files changed, 465 insertions(+), 1634 deletions(-)
> create mode 100644 tools/perf/pmu-events/empty-pmu-events.c
> delete mode 100644 tools/perf/pmu-events/jevents.c
> create mode 100755 tools/perf/pmu-events/jevents.py
> delete mode 100644 tools/perf/pmu-events/jsmn.h
> delete mode 100644 tools/perf/pmu-events/json.c
> delete mode 100644 tools/perf/pmu-events/json.h
>
--
Zhengjun Xing
On Fri, Jun 17, 2022 at 5:23 PM Xing Zhengjun
<[email protected]> wrote:
>
> Hi Ian,
>
> On 6/18/2022 8:09 AM, Ian Rogers wrote:
> > New architectures bring new complexity, such as Intel's hybrid
> > models. jevents provides an alternative to specifying events in the
> > kernel and exposing them through sysfs, however, it is difficult to
> > work with. For example, an error in the json input would yield an
> > error message but no json file or location. It is also a challenge to
> > update jsmn.c given its forked nature.
> >
> > The changes here switch from jevents.c to a rewrite in python called
> > jevents.py. This means there is a build time dependency on python, but
> > such a dependency already exists for asciidoc (used to generate perf's
> > man pages). If the build detects that python isn't present or is older
> > than version 3.6 (released Dec. 2016) then an empty file is
> > substituted for the generated one.
> >
> > A challenge with this code is in avoiding regressions. For this reason
> > the jevents.py produces identical output to jevents.c, validated with a
> > test script and build target.
> >
> > v6. Fixes an annotation and use of removesuffix that aren't present in
> > Python 3.6. Linter issues are also fixed.The code was tested on
> > Python 3.6 and 3.8 with docker.
>
> I just test it, now it can be built successfully with python 3.8.
Great! Tested-by/Reviewed-by/Acked-by always welcome :-)
Ian
> > v5. Adds a 2>/dev/null as suggested by David Laight
> > <[email protected]>.
> > v4. Fixes the build for systems with python2 installed by adding a
> > patch that makes python3 the preferred python (Reported-by: John
> > Garry <[email protected]>). It also fixes a bash-ism in the
> > jevents-test.sh and fixes the handling of an empty string for a
> > metric BriefDescription as one was added for sapphirerapids in the
> > metric Execute.
> > v3. Updates the patches for merged changes (on
> > acme/tmp.perf/core). Re-runs all comparisons to make sure the
> > generated pmu-events.c isn't altered at all by this change. Adds
> > the jevents.c ExtSel fix in:
> > https://lore.kernel.org/lkml/[email protected]/
> > Bumps the python version from 3.5 to 3.6, as f-strings weren't
> > introduced until 3.6.
> >
> > v2. Fixes the build for architectures that don't have pmu-events json
> > (Suggested-by: John Garry <[email protected]>) and fixes the
> > build for python not being present or too old (Suggested-by: Peter
> > Zijlstra <[email protected]>/John Garry <[email protected]>).
> >
> > Ian Rogers (4):
> > perf python: Prefer python3
> > perf jevents: Add python converter script
> > perf jevents: Switch build to use jevents.py
> > perf jevents: Remove jevents.c
> >
> > tools/perf/Makefile.config | 27 +-
> > tools/perf/Makefile.perf | 16 +-
> > tools/perf/pmu-events/Build | 15 +-
> > tools/perf/pmu-events/empty-pmu-events.c | 21 +
> > tools/perf/pmu-events/jevents.c | 1342 ----------------------
> > tools/perf/pmu-events/jevents.py | 409 +++++++
> > tools/perf/pmu-events/jsmn.h | 68 --
> > tools/perf/pmu-events/json.c | 162 ---
> > tools/perf/pmu-events/json.h | 39 -
> > 9 files changed, 465 insertions(+), 1634 deletions(-)
> > create mode 100644 tools/perf/pmu-events/empty-pmu-events.c
> > delete mode 100644 tools/perf/pmu-events/jevents.c
> > create mode 100755 tools/perf/pmu-events/jevents.py
> > delete mode 100644 tools/perf/pmu-events/jsmn.h
> > delete mode 100644 tools/perf/pmu-events/json.c
> > delete mode 100644 tools/perf/pmu-events/json.h
> >
>
> --
> Zhengjun Xing
Hi Ian,
On 6/18/2022 9:41 AM, Ian Rogers wrote:
> On Fri, Jun 17, 2022 at 5:23 PM Xing Zhengjun
> <[email protected]> wrote:
>>
>> Hi Ian,
>>
>> On 6/18/2022 8:09 AM, Ian Rogers wrote:
>>> New architectures bring new complexity, such as Intel's hybrid
>>> models. jevents provides an alternative to specifying events in the
>>> kernel and exposing them through sysfs, however, it is difficult to
>>> work with. For example, an error in the json input would yield an
>>> error message but no json file or location. It is also a challenge to
>>> update jsmn.c given its forked nature.
>>>
>>> The changes here switch from jevents.c to a rewrite in python called
>>> jevents.py. This means there is a build time dependency on python, but
>>> such a dependency already exists for asciidoc (used to generate perf's
>>> man pages). If the build detects that python isn't present or is older
>>> than version 3.6 (released Dec. 2016) then an empty file is
>>> substituted for the generated one.
>>>
>>> A challenge with this code is in avoiding regressions. For this reason
>>> the jevents.py produces identical output to jevents.c, validated with a
>>> test script and build target.
>>>
>>> v6. Fixes an annotation and use of removesuffix that aren't present in
>>> Python 3.6. Linter issues are also fixed.The code was tested on
>>> Python 3.6 and 3.8 with docker.
>>
>> I just test it, now it can be built successfully with python 3.8.
>
> Great! Tested-by/Reviewed-by/Acked-by always welcome :-)
I test python3.8 and python3.10, both can build successfully.
I also check the "pmu-events.c" generated by both C and python code,
they are the same.
Tested-by: Zhengjun Xing <[email protected]>
>
> Ian
>
>>> v5. Adds a 2>/dev/null as suggested by David Laight
>>> <[email protected]>.
>>> v4. Fixes the build for systems with python2 installed by adding a
>>> patch that makes python3 the preferred python (Reported-by: John
>>> Garry <[email protected]>). It also fixes a bash-ism in the
>>> jevents-test.sh and fixes the handling of an empty string for a
>>> metric BriefDescription as one was added for sapphirerapids in the
>>> metric Execute.
>>> v3. Updates the patches for merged changes (on
>>> acme/tmp.perf/core). Re-runs all comparisons to make sure the
>>> generated pmu-events.c isn't altered at all by this change. Adds
>>> the jevents.c ExtSel fix in:
>>> https://lore.kernel.org/lkml/[email protected]/
>>> Bumps the python version from 3.5 to 3.6, as f-strings weren't
>>> introduced until 3.6.
>>>
>>> v2. Fixes the build for architectures that don't have pmu-events json
>>> (Suggested-by: John Garry <[email protected]>) and fixes the
>>> build for python not being present or too old (Suggested-by: Peter
>>> Zijlstra <[email protected]>/John Garry <[email protected]>).
>>>
>>> Ian Rogers (4):
>>> perf python: Prefer python3
>>> perf jevents: Add python converter script
>>> perf jevents: Switch build to use jevents.py
>>> perf jevents: Remove jevents.c
>>>
>>> tools/perf/Makefile.config | 27 +-
>>> tools/perf/Makefile.perf | 16 +-
>>> tools/perf/pmu-events/Build | 15 +-
>>> tools/perf/pmu-events/empty-pmu-events.c | 21 +
>>> tools/perf/pmu-events/jevents.c | 1342 ----------------------
>>> tools/perf/pmu-events/jevents.py | 409 +++++++
>>> tools/perf/pmu-events/jsmn.h | 68 --
>>> tools/perf/pmu-events/json.c | 162 ---
>>> tools/perf/pmu-events/json.h | 39 -
>>> 9 files changed, 465 insertions(+), 1634 deletions(-)
>>> create mode 100644 tools/perf/pmu-events/empty-pmu-events.c
>>> delete mode 100644 tools/perf/pmu-events/jevents.c
>>> create mode 100755 tools/perf/pmu-events/jevents.py
>>> delete mode 100644 tools/perf/pmu-events/jsmn.h
>>> delete mode 100644 tools/perf/pmu-events/json.c
>>> delete mode 100644 tools/perf/pmu-events/json.h
>>>
>>
>> --
>> Zhengjun Xing
--
Zhengjun Xing
On Fri, Jun 17, 2022 at 05:09:03PM -0700, Ian Rogers wrote:
> New architectures bring new complexity, such as Intel's hybrid
> models. jevents provides an alternative to specifying events in the
> kernel and exposing them through sysfs, however, it is difficult to
> work with. For example, an error in the json input would yield an
> error message but no json file or location. It is also a challenge to
> update jsmn.c given its forked nature.
>
> The changes here switch from jevents.c to a rewrite in python called
> jevents.py. This means there is a build time dependency on python, but
> such a dependency already exists for asciidoc (used to generate perf's
> man pages). If the build detects that python isn't present or is older
> than version 3.6 (released Dec. 2016) then an empty file is
> substituted for the generated one.
>
> A challenge with this code is in avoiding regressions. For this reason
> the jevents.py produces identical output to jevents.c, validated with a
> test script and build target.
>
> v6. Fixes an annotation and use of removesuffix that aren't present in
> Python 3.6. Linter issues are also fixed.The code was tested on
> Python 3.6 and 3.8 with docker.
hi,
I'm getting test failures with this:
10: PMU events :
10.1: PMU event table sanity : FAILED!
10.2: PMU event map aliases : FAILED!
10.3: Parsing of PMU event table metrics : Ok
10.4: Parsing of PMU event table metrics with fake PMUs : Ok
I can see that pass without the patchset, verbose run shows:
[jolsa@krava perf]$ sudo ./perf test -vv 'PMU event map aliases'
10: PMU events :
10.2: PMU event map aliases :
--- start ---
test child forked, pid 272763
Using CPUID GenuineIntel-6-8C-1
intel_pt default config: tsc,mtc,mtc_period=3,psb_period=3,pt,branch
could not find test events map
testing core PMU cpu aliases: failed
test child finished with -1
---- end ----
PMU events subtest 2: FAILED!
[jolsa@krava perf]$ sudo ./perf test -vv 'PMU event table sanity'
10: PMU events :
10.1: PMU event table sanity :
--- start ---
test child forked, pid 272771
could not find test events map
test child finished with -1
---- end ----
PMU events subtest 1: FAILED!
jirka
> v5. Adds a 2>/dev/null as suggested by David Laight
> <[email protected]>.
> v4. Fixes the build for systems with python2 installed by adding a
> patch that makes python3 the preferred python (Reported-by: John
> Garry <[email protected]>). It also fixes a bash-ism in the
> jevents-test.sh and fixes the handling of an empty string for a
> metric BriefDescription as one was added for sapphirerapids in the
> metric Execute.
> v3. Updates the patches for merged changes (on
> acme/tmp.perf/core). Re-runs all comparisons to make sure the
> generated pmu-events.c isn't altered at all by this change. Adds
> the jevents.c ExtSel fix in:
> https://lore.kernel.org/lkml/[email protected]/
> Bumps the python version from 3.5 to 3.6, as f-strings weren't
> introduced until 3.6.
>
> v2. Fixes the build for architectures that don't have pmu-events json
> (Suggested-by: John Garry <[email protected]>) and fixes the
> build for python not being present or too old (Suggested-by: Peter
> Zijlstra <[email protected]>/John Garry <[email protected]>).
>
> Ian Rogers (4):
> perf python: Prefer python3
> perf jevents: Add python converter script
> perf jevents: Switch build to use jevents.py
> perf jevents: Remove jevents.c
>
> tools/perf/Makefile.config | 27 +-
> tools/perf/Makefile.perf | 16 +-
> tools/perf/pmu-events/Build | 15 +-
> tools/perf/pmu-events/empty-pmu-events.c | 21 +
> tools/perf/pmu-events/jevents.c | 1342 ----------------------
> tools/perf/pmu-events/jevents.py | 409 +++++++
> tools/perf/pmu-events/jsmn.h | 68 --
> tools/perf/pmu-events/json.c | 162 ---
> tools/perf/pmu-events/json.h | 39 -
> 9 files changed, 465 insertions(+), 1634 deletions(-)
> create mode 100644 tools/perf/pmu-events/empty-pmu-events.c
> delete mode 100644 tools/perf/pmu-events/jevents.c
> create mode 100755 tools/perf/pmu-events/jevents.py
> delete mode 100644 tools/perf/pmu-events/jsmn.h
> delete mode 100644 tools/perf/pmu-events/json.c
> delete mode 100644 tools/perf/pmu-events/json.h
>
> --
> 2.36.1.476.g0c4daa206d-goog
>
On Wed, Jun 22, 2022 at 09:12:40AM +0200, Jiri Olsa wrote:
> On Fri, Jun 17, 2022 at 05:09:03PM -0700, Ian Rogers wrote:
> > New architectures bring new complexity, such as Intel's hybrid
> > models. jevents provides an alternative to specifying events in the
> > kernel and exposing them through sysfs, however, it is difficult to
> > work with. For example, an error in the json input would yield an
> > error message but no json file or location. It is also a challenge to
> > update jsmn.c given its forked nature.
> >
> > The changes here switch from jevents.c to a rewrite in python called
> > jevents.py. This means there is a build time dependency on python, but
> > such a dependency already exists for asciidoc (used to generate perf's
> > man pages). If the build detects that python isn't present or is older
> > than version 3.6 (released Dec. 2016) then an empty file is
> > substituted for the generated one.
> >
> > A challenge with this code is in avoiding regressions. For this reason
> > the jevents.py produces identical output to jevents.c, validated with a
> > test script and build target.
> >
> > v6. Fixes an annotation and use of removesuffix that aren't present in
> > Python 3.6. Linter issues are also fixed.The code was tested on
> > Python 3.6 and 3.8 with docker.
>
> hi,
> I'm getting test failures with this:
>
> 10: PMU events :
> 10.1: PMU event table sanity : FAILED!
> 10.2: PMU event map aliases : FAILED!
> 10.3: Parsing of PMU event table metrics : Ok
> 10.4: Parsing of PMU event table metrics with fake PMUs : Ok
>
> I can see that pass without the patchset, verbose run shows:
>
> [jolsa@krava perf]$ sudo ./perf test -vv 'PMU event map aliases'
> 10: PMU events :
> 10.2: PMU event map aliases :
> --- start ---
> test child forked, pid 272763
> Using CPUID GenuineIntel-6-8C-1
> intel_pt default config: tsc,mtc,mtc_period=3,psb_period=3,pt,branch
> could not find test events map
> testing core PMU cpu aliases: failed
> test child finished with -1
> ---- end ----
> PMU events subtest 2: FAILED!
> [jolsa@krava perf]$ sudo ./perf test -vv 'PMU event table sanity'
> 10: PMU events :
> 10.1: PMU event table sanity :
> --- start ---
> test child forked, pid 272771
> could not find test events map
> test child finished with -1
> ---- end ----
> PMU events subtest 1: FAILED!
ah, ok I did not notice:
Makefile.config:909: Python interpreter too old (older than 3.6) disabling jevent generation
I installed python3-devel and it's ok
should we make above tests SKIP in case there's empty pmu-events.c ?
thanks,
jirka
On Wed, Jun 22, 2022 at 12:36 AM Jiri Olsa <[email protected]> wrote:
>
> On Wed, Jun 22, 2022 at 09:12:40AM +0200, Jiri Olsa wrote:
> > On Fri, Jun 17, 2022 at 05:09:03PM -0700, Ian Rogers wrote:
> > > New architectures bring new complexity, such as Intel's hybrid
> > > models. jevents provides an alternative to specifying events in the
> > > kernel and exposing them through sysfs, however, it is difficult to
> > > work with. For example, an error in the json input would yield an
> > > error message but no json file or location. It is also a challenge to
> > > update jsmn.c given its forked nature.
> > >
> > > The changes here switch from jevents.c to a rewrite in python called
> > > jevents.py. This means there is a build time dependency on python, but
> > > such a dependency already exists for asciidoc (used to generate perf's
> > > man pages). If the build detects that python isn't present or is older
> > > than version 3.6 (released Dec. 2016) then an empty file is
> > > substituted for the generated one.
> > >
> > > A challenge with this code is in avoiding regressions. For this reason
> > > the jevents.py produces identical output to jevents.c, validated with a
> > > test script and build target.
> > >
> > > v6. Fixes an annotation and use of removesuffix that aren't present in
> > > Python 3.6. Linter issues are also fixed.The code was tested on
> > > Python 3.6 and 3.8 with docker.
> >
> > hi,
> > I'm getting test failures with this:
> >
> > 10: PMU events :
> > 10.1: PMU event table sanity : FAILED!
> > 10.2: PMU event map aliases : FAILED!
> > 10.3: Parsing of PMU event table metrics : Ok
> > 10.4: Parsing of PMU event table metrics with fake PMUs : Ok
> >
> > I can see that pass without the patchset, verbose run shows:
> >
> > [jolsa@krava perf]$ sudo ./perf test -vv 'PMU event map aliases'
> > 10: PMU events :
> > 10.2: PMU event map aliases :
> > --- start ---
> > test child forked, pid 272763
> > Using CPUID GenuineIntel-6-8C-1
> > intel_pt default config: tsc,mtc,mtc_period=3,psb_period=3,pt,branch
> > could not find test events map
> > testing core PMU cpu aliases: failed
> > test child finished with -1
> > ---- end ----
> > PMU events subtest 2: FAILED!
> > [jolsa@krava perf]$ sudo ./perf test -vv 'PMU event table sanity'
> > 10: PMU events :
> > 10.1: PMU event table sanity :
> > --- start ---
> > test child forked, pid 272771
> > could not find test events map
> > test child finished with -1
> > ---- end ----
> > PMU events subtest 1: FAILED!
>
> ah, ok I did not notice:
>
> Makefile.config:909: Python interpreter too old (older than 3.6) disabling jevent generation
>
> I installed python3-devel and it's ok
>
> should we make above tests SKIP in case there's empty pmu-events.c ?
Hmm.. I didn't know the empty case was failing, that seems
sub-optimal. It'd also be nice to report here a skip for not being
compiled in rather than a pass. Let me take a look later.
Thanks,
Ian
> thanks,
> jirka