New architectures bring new complexity, such as Intel's hybrid
models. jevents provides an alternative to specifying events in the
kernel and exposing them through sysfs, however, it is difficult to
work with. For example, an error in the json input would yield an
error message but no json file or location. It is also a challenge to
update jsmn.c given its forked nature.
The changes here switch from jevents.c to a rewrite in python called
jevents.py. This means there is a build time dependency on python, but
such a dependency already exists for asciidoc (used to generate perf's
man pages). If the build detects that python isn't present or is older
than version 3.6 (released Dec. 2016) then an empty file is
substituted for the generated one.
A challenge with this code is in avoiding regressions. For this reason
the jevents.py produces identical output to jevents.c, validated with a
test script and build target.
v3. Updates the patches for merged changes (on
acme/tmp.perf/core). Re-runs all comparisons to make sure the
generated pmu-events.c isn't altered at all by this change. Adds
the jevents.c ExtSel fix in:
https://lore.kernel.org/lkml/[email protected]/
Bumps the python version from 3.5 to 3.6, as f-strings weren't
introduced until 3.6.
v2. Fixes the build for architectures that don't have pmu-events json
(Suggested-by: John Garry <[email protected]>) and fixes the
build for python not being present or too old (Suggested-by: Peter
Zijlstra <[email protected]>/John Garry <[email protected]>).
Ian Rogers (3):
perf jevents: Add python converter script
perf jevents: Switch build to use jevents.py
perf jevents: Remove jevents.c
tools/perf/Makefile.config | 19 +
tools/perf/Makefile.perf | 16 +-
tools/perf/pmu-events/Build | 15 +-
tools/perf/pmu-events/empty-pmu-events.c | 21 +
tools/perf/pmu-events/jevents.c | 1342 ----------------------
tools/perf/pmu-events/jevents.py | 392 +++++++
tools/perf/pmu-events/jsmn.h | 68 --
tools/perf/pmu-events/json.c | 162 ---
tools/perf/pmu-events/json.h | 39 -
9 files changed, 444 insertions(+), 1630 deletions(-)
create mode 100644 tools/perf/pmu-events/empty-pmu-events.c
delete mode 100644 tools/perf/pmu-events/jevents.c
create mode 100755 tools/perf/pmu-events/jevents.py
delete mode 100644 tools/perf/pmu-events/jsmn.h
delete mode 100644 tools/perf/pmu-events/json.c
delete mode 100644 tools/perf/pmu-events/json.h
--
2.36.1.255.ge46751e96f-goog
On Wed, Jun 1, 2022 at 9:17 AM John Garry <[email protected]> wrote:
>
> On 27/05/2022 19:54, Ian Rogers wrote:
> > New architectures bring new complexity, such as Intel's hybrid
> > models. jevents provides an alternative to specifying events in the
> > kernel and exposing them through sysfs, however, it is difficult to
> > work with. For example, an error in the json input would yield an
> > error message but no json file or location. It is also a challenge to
> > update jsmn.c given its forked nature.
> >
> > The changes here switch from jevents.c to a rewrite in python called
> > jevents.py. This means there is a build time dependency on python, but
> > such a dependency already exists for asciidoc (used to generate perf's
> > man pages). If the build detects that python isn't present or is older
> > than version 3.6 (released Dec. 2016) then an empty file is
> > substituted for the generated one.
> >
> > A challenge with this code is in avoiding regressions. For this reason
> > the jevents.py produces identical output to jevents.c, validated with a
> > test script and build target.
> >
>
> Hi Ian,
>
> I still see this:
>
> ...
> Makefile.config:906: Python interpreter too old (older than 3.6)
> disabling jevent generation
> Makefile.config:939: Old version of libbfd/binutils things like PE
> executable profiling will not be available
> Makefile.config:1127: No openjdk development package found, please
> install JDK package, e.g. openjdk-8-jdk, java-1.8.0-openjdk-devel
>
> Auto-detecting system features:
> ... dwarf: [ on ]
> ... dwarf_getlocations: [ on ]
> ... glibc: [ on ]
> ... libbfd: [ OFF ]
> ... libbfd-buildid: [ OFF ]
> ... libcap: [ on ]
> ... libelf: [ on ]
> ... libnuma: [ on ]
> ... numa_num_possible_cpus: [ on ]
> ... libperl: [ on ]
> ... libpython: [ on ]
> ... libcrypto: [ on ]
> ... libunwind: [ on ]
> ... libdw-dwarf-unwind: [ on ]
> ... zlib: [ on ]
> ... lzma: [ on ]
> ... get_cpuid: [ on ]
> ... bpf: [ on ]
> ... libaio: [ on ]
> ... libzstd: [ on ]
> ... disassembler-four-args: [ on ]
>
>
> make[3]: Nothing to be done for 'install_headers'.
> john@localhost:~/acme/tools/perf> python --version
> Python 3.6.12
>
> I need to figure out what is going wrong...
Thanks for testing John! Could you try adding something like this:
```
diff --git a/tools/perf/Makefile.config b/tools/perf/Makefile.config
index 9538d737d655..c23f39e21a70 100644
--- a/tools/perf/Makefile.config
+++ b/tools/perf/Makefile.config
@@ -901,6 +901,8 @@ ifneq ($(NO_JEVENTS),1)
NO_JEVENTS := 1
else
# jevents.py uses f-strings present in Python 3.6 released in Dec. 2016.
+ PYTHON_VERSION=$(shell $(PYTHON) -c 'import sys;print(sys.version_info)')
+ $(error $(PYTHON_VERSION))
JEVENTS_PYTHON_GOOD := $(shell $(PYTHON) -c 'import sys;print("1"
if(sys.version_info.major >=
3 and sys.version_info.minor >= 6) else "0")')
ifneq ($(JEVENTS_PYTHON_GOOD), 1)
$(warning Python interpreter too old (older than 3.6) disabling
jevent generation)
```
You can also specify which python you want on the make command line,
for example adding PYTHON=`which python` at the end of your make
command.
Thanks,
Ian
> Thanks,
> John
>
> > v3. Updates the patches for merged changes (on
> > acme/tmp.perf/core). Re-runs all comparisons to make sure the
> > generated pmu-events.c isn't altered at all by this change. Adds
> > the jevents.c ExtSel fix in:
> > https://lore.kernel.org/lkml/[email protected]/
> > Bumps the python version from 3.5 to 3.6, as f-strings weren't
> > introduced until 3.6.
> >
> > v2. Fixes the build for architectures that don't have pmu-events json
> > (Suggested-by: John Garry <[email protected]>) and fixes the
> > build for python not being present or too old (Suggested-by: Peter
> > Zijlstra <[email protected]>/John Garry <[email protected]>).
> >
> > Ian Rogers (3):
> > perf jevents: Add python converter script
> > perf jevents: Switch build to use jevents.py
> > perf jevents: Remove jevents.c
> >
> > tools/perf/Makefile.config | 19 +
> > tools/perf/Makefile.perf | 16 +-
> > tools/perf/pmu-events/Build | 15 +-
> > tools/perf/pmu-events/empty-pmu-events.c | 21 +
> > tools/perf/pmu-events/jevents.c | 1342 ----------------------
> > tools/perf/pmu-events/jevents.py | 392 +++++++
> > tools/perf/pmu-events/jsmn.h | 68 --
> > tools/perf/pmu-events/json.c | 162 ---
> > tools/perf/pmu-events/json.h | 39 -
> > 9 files changed, 444 insertions(+), 1630 deletions(-)
> > create mode 100644 tools/perf/pmu-events/empty-pmu-events.c
> > delete mode 100644 tools/perf/pmu-events/jevents.c
> > create mode 100755 tools/perf/pmu-events/jevents.py
> > delete mode 100644 tools/perf/pmu-events/jsmn.h
> > delete mode 100644 tools/perf/pmu-events/json.c
> > delete mode 100644 tools/perf/pmu-events/json.h
> >
>
On Tue, Jun 14, 2022 at 8:50 AM John Garry <[email protected]> wrote:
>
> On 01/06/2022 23:43, Ian Rogers wrote:
> > On Wed, Jun 1, 2022 at 9:17 AM John Garry<[email protected]> wrote:
> >> On 27/05/2022 19:54, Ian Rogers wrote:
> >>> New architectures bring new complexity, such as Intel's hybrid
> >>> models. jevents provides an alternative to specifying events in the
> >>> kernel and exposing them through sysfs, however, it is difficult to
> >>> work with. For example, an error in the json input would yield an
> >>> error message but no json file or location. It is also a challenge to
> >>> update jsmn.c given its forked nature.
> >>>
> >>> The changes here switch from jevents.c to a rewrite in python called
> >>> jevents.py. This means there is a build time dependency on python, but
> >>> such a dependency already exists for asciidoc (used to generate perf's
> >>> man pages). If the build detects that python isn't present or is older
> >>> than version 3.6 (released Dec. 2016) then an empty file is
> >>> substituted for the generated one.
> >>>
> >>> A challenge with this code is in avoiding regressions. For this reason
> >>> the jevents.py produces identical output to jevents.c, validated with a
> >>> test script and build target.
> >>>
> >> Hi Ian,
> >>
> >> I still see this:
> >>
> >> ...
> >> Makefile.config:906: Python interpreter too old (older than 3.6)
> >> disabling jevent generation
> >> Makefile.config:939: Old version of libbfd/binutils things like PE
> >> executable profiling will not be available
> >> Makefile.config:1127: No openjdk development package found, please
> >> install JDK package, e.g. openjdk-8-jdk, java-1.8.0-openjdk-devel
> >>
> >> Auto-detecting system features:
> >> ... dwarf: [ on ]
> >> ... dwarf_getlocations: [ on ]
> >> ... glibc: [ on ]
> >> ... libbfd: [ OFF ]
> >> ... libbfd-buildid: [ OFF ]
> >> ... libcap: [ on ]
> >> ... libelf: [ on ]
> >> ... libnuma: [ on ]
> >> ... numa_num_possible_cpus: [ on ]
> >> ... libperl: [ on ]
> >> ... libpython: [ on ]
> >> ... libcrypto: [ on ]
> >> ... libunwind: [ on ]
> >> ... libdw-dwarf-unwind: [ on ]
> >> ... zlib: [ on ]
> >> ... lzma: [ on ]
> >> ... get_cpuid: [ on ]
> >> ... bpf: [ on ]
> >> ... libaio: [ on ]
> >> ... libzstd: [ on ]
> >> ... disassembler-four-args: [ on ]
> >>
> >>
> >> make[3]: Nothing to be done for 'install_headers'.
> >> john@localhost:~/acme/tools/perf> python --version
> >> Python 3.6.12
> >>
> >> I need to figure out what is going wrong...
> > Thanks for testing John! Could you try adding something like this:
> > ```
> > diff --git a/tools/perf/Makefile.config b/tools/perf/Makefile.config
> > index 9538d737d655..c23f39e21a70 100644
> > --- a/tools/perf/Makefile.config
> > +++ b/tools/perf/Makefile.config
> > @@ -901,6 +901,8 @@ ifneq ($(NO_JEVENTS),1)
> > NO_JEVENTS := 1
> > else
> > # jevents.py uses f-strings present in Python 3.6 released in Dec. 2016.
> > + PYTHON_VERSION=$(shell $(PYTHON) -c 'import sys;print(sys.version_info)')
> > + $(error $(PYTHON_VERSION))
> > JEVENTS_PYTHON_GOOD := $(shell $(PYTHON) -c 'import sys;print("1"
> > if(sys.version_info.major >=
> > 3 and sys.version_info.minor >= 6) else "0")')
> > ifneq ($(JEVENTS_PYTHON_GOOD), 1)
> > $(warning Python interpreter too old (older than 3.6) disabling
> > jevent generation)
> > ```
>
> That gives:
>
> Makefile.config:459: No libdw DWARF unwind found, Please install
> elfutils-devel/libdw-dev >= 0.158 and/or set LIBDW_DIR
> Makefile.config:464: No libdw.h found or old libdw.h found or elfutils
> is older than 0.138, disables dwarf support. Please install new
> elfutils-devel/libdw-dev
> Makefile.config:617: DWARF support is off, BPF prologue is disabled
> Makefile.config:667: No debug_frame support found in libunwind-aarch64
> Makefile.config:738: No debug_frame support found in libunwind
> Makefile.config:906: *** sys.version_info(major=2, minor=7, micro=13,
> releaselevel='final', serial=0).Stop.
> Makefile.perf:239: recipe for target 'sub-make' failed
> make[1]: *** [sub-make] Error 2
> Makefile:69: recipe for target 'all' failed
> make: *** [all] Error 2
> john@debian:~/linux/tools/perf$ python --version
> Python 3.6.9
> john@debian:~/linux/tools/perf$
>
> > You can also specify which python you want on the make command line,
> > for example adding PYTHON=`which python` at the end of your make
> > command.
>
> diff -u tools/perf/trace/beauty/include/linux/socket.h
> include/linux/socket.h
> Makefile.config:259: *** which python-config not found. Stop.
> Makefile.perf:239: recipe for target 'sub-make' failed
> make[1]: *** [sub-make] Error 2
> Makefile:69: recipe for target 'all' failed
> make: *** [all] Error 2
> john@debian:~/linux/tools/perf$
>
> JFYI, this is my alternatives:
>
> sudo update-alternatives --config python
> There are 3 choices for the alternative python (providing
> /usr/bin/python).
>
> Selection Path Priority Status
> ------------------------------------------------------------
> 0 /usr/local/bin/python3.6 50 auto mode
> 1 /usr/bin/python2.7 1 manual mode
> 2 /usr/bin/python3.5 2 manual mode
> * 3 /usr/local/bin/python3.6 50 manual mode
>
> Press <enter> to keep the current choice[*], or type selection number:
Thanks Gary,
Perhaps it relates to dev packages. Perhaps, apt-get install
python-dev-is-python3 which should resolve the symlinks. I wonder that
you have python dev for python2 but python3 installed without the dev.
We should be able to make a Makefile test for this.
Thanks,
Ian
>
>
> Can someone else try this? Arnaldo?
>
> Sorry for being useless with this....
On 14/06/2022 17:03, Ian Rogers wrote:
>> diff -u tools/perf/trace/beauty/include/linux/socket.h
>> include/linux/socket.h
>> Makefile.config:259: *** which python-config not found. Stop.
>> Makefile.perf:239: recipe for target 'sub-make' failed
>> make[1]: *** [sub-make] Error 2
>> Makefile:69: recipe for target 'all' failed
>> make: *** [all] Error 2
>> john@debian:~/linux/tools/perf$
>>
>> JFYI, this is my alternatives:
>>
>> sudo update-alternatives --config python
>> There are 3 choices for the alternative python (providing
>> /usr/bin/python).
>>
>> Selection Path Priority Status
>> ------------------------------------------------------------
>> 0 /usr/local/bin/python3.6 50 auto mode
>> 1 /usr/bin/python2.7 1 manual mode
>> 2 /usr/bin/python3.5 2 manual mode
>> * 3 /usr/local/bin/python3.6 50 manual mode
>>
>> Press <enter> to keep the current choice[*], or type selection number:
> Thanks Gary,
>
> Perhaps it relates to dev packages. Perhaps, apt-get install
> python-dev-is-python3 which should resolve the symlinks. I wonder that
> you have python dev for python2 but python3 installed without the dev.
> We should be able to make a Makefile test for this.
So I was trying another system here as a sanity test (as my main dev box
is not working either). And this other system is debian and only
supports python up to 3.5, so that is why I have
/usr/local/bin/python3.6 as an alt, i.e. I downloaded and built myself.
Anyway, I do have python3-dev:
john@debian:~/linux/tools/perf$ sudo apt-get install python3-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
python3-dev is already the newest version (3.5.3-1).
On 01/06/2022 23:43, Ian Rogers wrote:
> On Wed, Jun 1, 2022 at 9:17 AM John Garry<[email protected]> wrote:
>> On 27/05/2022 19:54, Ian Rogers wrote:
>>> New architectures bring new complexity, such as Intel's hybrid
>>> models. jevents provides an alternative to specifying events in the
>>> kernel and exposing them through sysfs, however, it is difficult to
>>> work with. For example, an error in the json input would yield an
>>> error message but no json file or location. It is also a challenge to
>>> update jsmn.c given its forked nature.
>>>
>>> The changes here switch from jevents.c to a rewrite in python called
>>> jevents.py. This means there is a build time dependency on python, but
>>> such a dependency already exists for asciidoc (used to generate perf's
>>> man pages). If the build detects that python isn't present or is older
>>> than version 3.6 (released Dec. 2016) then an empty file is
>>> substituted for the generated one.
>>>
>>> A challenge with this code is in avoiding regressions. For this reason
>>> the jevents.py produces identical output to jevents.c, validated with a
>>> test script and build target.
>>>
>> Hi Ian,
>>
>> I still see this:
>>
>> ...
>> Makefile.config:906: Python interpreter too old (older than 3.6)
>> disabling jevent generation
>> Makefile.config:939: Old version of libbfd/binutils things like PE
>> executable profiling will not be available
>> Makefile.config:1127: No openjdk development package found, please
>> install JDK package, e.g. openjdk-8-jdk, java-1.8.0-openjdk-devel
>>
>> Auto-detecting system features:
>> ... dwarf: [ on ]
>> ... dwarf_getlocations: [ on ]
>> ... glibc: [ on ]
>> ... libbfd: [ OFF ]
>> ... libbfd-buildid: [ OFF ]
>> ... libcap: [ on ]
>> ... libelf: [ on ]
>> ... libnuma: [ on ]
>> ... numa_num_possible_cpus: [ on ]
>> ... libperl: [ on ]
>> ... libpython: [ on ]
>> ... libcrypto: [ on ]
>> ... libunwind: [ on ]
>> ... libdw-dwarf-unwind: [ on ]
>> ... zlib: [ on ]
>> ... lzma: [ on ]
>> ... get_cpuid: [ on ]
>> ... bpf: [ on ]
>> ... libaio: [ on ]
>> ... libzstd: [ on ]
>> ... disassembler-four-args: [ on ]
>>
>>
>> make[3]: Nothing to be done for 'install_headers'.
>> john@localhost:~/acme/tools/perf> python --version
>> Python 3.6.12
>>
>> I need to figure out what is going wrong...
> Thanks for testing John! Could you try adding something like this:
> ```
> diff --git a/tools/perf/Makefile.config b/tools/perf/Makefile.config
> index 9538d737d655..c23f39e21a70 100644
> --- a/tools/perf/Makefile.config
> +++ b/tools/perf/Makefile.config
> @@ -901,6 +901,8 @@ ifneq ($(NO_JEVENTS),1)
> NO_JEVENTS := 1
> else
> # jevents.py uses f-strings present in Python 3.6 released in Dec. 2016.
> + PYTHON_VERSION=$(shell $(PYTHON) -c 'import sys;print(sys.version_info)')
> + $(error $(PYTHON_VERSION))
> JEVENTS_PYTHON_GOOD := $(shell $(PYTHON) -c 'import sys;print("1"
> if(sys.version_info.major >=
> 3 and sys.version_info.minor >= 6) else "0")')
> ifneq ($(JEVENTS_PYTHON_GOOD), 1)
> $(warning Python interpreter too old (older than 3.6) disabling
> jevent generation)
> ```
That gives:
Makefile.config:459: No libdw DWARF unwind found, Please install
elfutils-devel/libdw-dev >= 0.158 and/or set LIBDW_DIR
Makefile.config:464: No libdw.h found or old libdw.h found or elfutils
is older than 0.138, disables dwarf support. Please install new
elfutils-devel/libdw-dev
Makefile.config:617: DWARF support is off, BPF prologue is disabled
Makefile.config:667: No debug_frame support found in libunwind-aarch64
Makefile.config:738: No debug_frame support found in libunwind
Makefile.config:906: *** sys.version_info(major=2, minor=7, micro=13,
releaselevel='final', serial=0).Stop.
Makefile.perf:239: recipe for target 'sub-make' failed
make[1]: *** [sub-make] Error 2
Makefile:69: recipe for target 'all' failed
make: *** [all] Error 2
john@debian:~/linux/tools/perf$ python --version
Python 3.6.9
john@debian:~/linux/tools/perf$
> You can also specify which python you want on the make command line,
> for example adding PYTHON=`which python` at the end of your make
> command.
diff -u tools/perf/trace/beauty/include/linux/socket.h
include/linux/socket.h
Makefile.config:259: *** which python-config not found. Stop.
Makefile.perf:239: recipe for target 'sub-make' failed
make[1]: *** [sub-make] Error 2
Makefile:69: recipe for target 'all' failed
make: *** [all] Error 2
john@debian:~/linux/tools/perf$
JFYI, this is my alternatives:
sudo update-alternatives --config python
There are 3 choices for the alternative python (providing
/usr/bin/python).
Selection Path Priority Status
------------------------------------------------------------
0 /usr/local/bin/python3.6 50 auto mode
1 /usr/bin/python2.7 1 manual mode
2 /usr/bin/python3.5 2 manual mode
* 3 /usr/local/bin/python3.6 50 manual mode
Press <enter> to keep the current choice[*], or type selection number:
Can someone else try this? Arnaldo?
Sorry for being useless with this....
On Tue, Jun 14, 2022 at 9:10 AM John Garry <[email protected]> wrote:
>
> On 14/06/2022 17:03, Ian Rogers wrote:
> >> diff -u tools/perf/trace/beauty/include/linux/socket.h
> >> include/linux/socket.h
> >> Makefile.config:259: *** which python-config not found. Stop.
> >> Makefile.perf:239: recipe for target 'sub-make' failed
> >> make[1]: *** [sub-make] Error 2
> >> Makefile:69: recipe for target 'all' failed
> >> make: *** [all] Error 2
> >> john@debian:~/linux/tools/perf$
> >>
> >> JFYI, this is my alternatives:
> >>
> >> sudo update-alternatives --config python
> >> There are 3 choices for the alternative python (providing
> >> /usr/bin/python).
> >>
> >> Selection Path Priority Status
> >> ------------------------------------------------------------
> >> 0 /usr/local/bin/python3.6 50 auto mode
> >> 1 /usr/bin/python2.7 1 manual mode
> >> 2 /usr/bin/python3.5 2 manual mode
> >> * 3 /usr/local/bin/python3.6 50 manual mode
> >>
> >> Press <enter> to keep the current choice[*], or type selection number:
> > Thanks Gary,
> >
> > Perhaps it relates to dev packages. Perhaps, apt-get install
> > python-dev-is-python3 which should resolve the symlinks. I wonder that
> > you have python dev for python2 but python3 installed without the dev.
> > We should be able to make a Makefile test for this.
>
> So I was trying another system here as a sanity test (as my main dev box
> is not working either). And this other system is debian and only
> supports python up to 3.5, so that is why I have
> /usr/local/bin/python3.6 as an alt, i.e. I downloaded and built myself.
>
> Anyway, I do have python3-dev:
>
> john@debian:~/linux/tools/perf$ sudo apt-get install python3-dev
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> python3-dev is already the newest version (3.5.3-1).
>
It turned out to be something simple. The build is set up to prefer
python2 over python3, so if you have both you get python2. I've fixed
this now in v4 (PTAL):
https://lore.kernel.org/lkml/[email protected]/
I think we should remove python2 build support as it is confusing and
python2 doesn't support setuptools meaning this patch fails:
https://lore.kernel.org/lkml/[email protected]/
and without that patch python3 generates deprecated module warnings.
Thanks,
Ian