On Sun, Jan 17, 2021 at 12:58 PM Sedat Dilek <[email protected]> wrote:
>
> On Sun, Jan 17, 2021 at 12:42 PM Sedat Dilek <[email protected]> wrote:
> >
> > On Sun, Jan 17, 2021 at 12:23 PM Sedat Dilek <[email protected]> wrote:
> > >
> > > On Sun, Jan 17, 2021 at 11:53 AM Sedat Dilek <[email protected]> wrote:
> > > >
> > > > On Sun, Jan 17, 2021 at 11:44 AM Sedat Dilek <[email protected]> wrote:
> > > > >
> > > > > On Sat, Jan 16, 2021 at 9:23 PM Bill Wendling <[email protected]> wrote:
> > > > > >
> > > > > > On Sat, Jan 16, 2021 at 9:39 AM Sedat Dilek <[email protected]> wrote:
> > > > > > > On Sat, Jan 16, 2021 at 10:44 AM 'Bill Wendling' via Clang Built Linux
> > > > > > > <[email protected]> wrote:
> > > > > > > >
> > > > > > > > From: Sami Tolvanen <[email protected]>
> > > > > > > >
> > > > > > > > Enable the use of clang's Profile-Guided Optimization[1]. To generate a
> > > > > > > > profile, the kernel is instrumented with PGO counters, a representative
> > > > > > > > workload is run, and the raw profile data is collected from
> > > > > > > > /sys/kernel/debug/pgo/profraw.
> > > > > > > >
> > > > > > > > The raw profile data must be processed by clang's "llvm-profdata" tool
> > > > > > > > before it can be used during recompilation:
> > > > > > > >
> > > > > > > > $ cp /sys/kernel/debug/pgo/profraw vmlinux.profraw
> > > > > > > > $ llvm-profdata merge --output=vmlinux.profdata vmlinux.profraw
> > > > > > > >
> > > > > > > > Multiple raw profiles may be merged during this step.
> > > > > > > >
> > > > > > > > The data can now be used by the compiler:
> > > > > > > >
> > > > > > > > $ make LLVM=1 KCFLAGS=-fprofile-use=vmlinux.profdata ...
> > > > > > > >
> > > > > > > > This initial submission is restricted to x86, as that's the platform we
> > > > > > > > know works. This restriction can be lifted once other platforms have
> > > > > > > > been verified to work with PGO.
> > > > > > > >
> > > > > > > > Note that this method of profiling the kernel is clang-native, unlike
> > > > > > > > the clang support in kernel/gcov.
> > > > > > > >
> > > > > > > > [1] https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization
> > > > > > > >
> > > > > > > > Signed-off-by: Sami Tolvanen <[email protected]>
> > > > > > > > Co-developed-by: Bill Wendling <[email protected]>
> > > > > > > > Signed-off-by: Bill Wendling <[email protected]>
> > > > > > > > ---
> > > > > > > > v2: - Added "__llvm_profile_instrument_memop" based on Nathan Chancellor's
> > > > > > > > testing.
> > > > > > > > - Corrected documentation, re PGO flags when using LTO, based on Fangrui
> > > > > > > > Song's comments.
> > > > > > > > v3: - Added change log section based on Sedat Dilek's comments.
> > > > > > > > v4: - Remove non-x86 Makfile changes and se "hweight64" instead of using our
> > > > > > > > own popcount implementation, based on Nick Desaulniers's comment.
> > > > > > > > v5: - Correct padding calculation, discovered by Nathan Chancellor.
> > > > > > > > ---
> > > > > > > > Documentation/dev-tools/index.rst | 1 +
> > > > > > > > Documentation/dev-tools/pgo.rst | 127 +++++++++
> > > > > > > > MAINTAINERS | 9 +
> > > > > > > > Makefile | 3 +
> > > > > > > > arch/Kconfig | 1 +
> > > > > > > > arch/x86/Kconfig | 1 +
> > > > > > > > arch/x86/boot/Makefile | 1 +
> > > > > > > > arch/x86/boot/compressed/Makefile | 1 +
> > > > > > > > arch/x86/crypto/Makefile | 2 +
> > > > > > > > arch/x86/entry/vdso/Makefile | 1 +
> > > > > > > > arch/x86/kernel/vmlinux.lds.S | 2 +
> > > > > > > > arch/x86/platform/efi/Makefile | 1 +
> > > > > > > > arch/x86/purgatory/Makefile | 1 +
> > > > > > > > arch/x86/realmode/rm/Makefile | 1 +
> > > > > > > > arch/x86/um/vdso/Makefile | 1 +
> > > > > > > > drivers/firmware/efi/libstub/Makefile | 1 +
> > > > > > > > include/asm-generic/vmlinux.lds.h | 44 +++
> > > > > > > > kernel/Makefile | 1 +
> > > > > > > > kernel/pgo/Kconfig | 35 +++
> > > > > > > > kernel/pgo/Makefile | 5 +
> > > > > > > > kernel/pgo/fs.c | 382 ++++++++++++++++++++++++++
> > > > > > > > kernel/pgo/instrument.c | 185 +++++++++++++
> > > > > > > > kernel/pgo/pgo.h | 206 ++++++++++++++
> > > > > > > > scripts/Makefile.lib | 10 +
> > > > > > > > 24 files changed, 1022 insertions(+)
> > > > > > > > create mode 100644 Documentation/dev-tools/pgo.rst
> > > > > > > > create mode 100644 kernel/pgo/Kconfig
> > > > > > > > create mode 100644 kernel/pgo/Makefile
> > > > > > > > create mode 100644 kernel/pgo/fs.c
> > > > > > > > create mode 100644 kernel/pgo/instrument.c
> > > > > > > > create mode 100644 kernel/pgo/pgo.h
> > > > > > > >
> > > > > > > > diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/index.rst
> > > > > > > > index f7809c7b1ba9e..8d6418e858062 100644
> > > > > > > > --- a/Documentation/dev-tools/index.rst
> > > > > > > > +++ b/Documentation/dev-tools/index.rst
> > > > > > > > @@ -26,6 +26,7 @@ whole; patches welcome!
> > > > > > > > kgdb
> > > > > > > > kselftest
> > > > > > > > kunit/index
> > > > > > > > + pgo
> > > > > > > >
> > > > > > > >
> > > > > > > > .. only:: subproject and html
> > > > > > > > diff --git a/Documentation/dev-tools/pgo.rst b/Documentation/dev-tools/pgo.rst
> > > > > > > > new file mode 100644
> > > > > > > > index 0000000000000..b7f11d8405b73
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/Documentation/dev-tools/pgo.rst
> > > > > > > > @@ -0,0 +1,127 @@
> > > > > > > > +.. SPDX-License-Identifier: GPL-2.0
> > > > > > > > +
> > > > > > > > +===============================
> > > > > > > > +Using PGO with the Linux kernel
> > > > > > > > +===============================
> > > > > > > > +
> > > > > > > > +Clang's profiling kernel support (PGO_) enables profiling of the Linux kernel
> > > > > > > > +when building with Clang. The profiling data is exported via the ``pgo``
> > > > > > > > +debugfs directory.
> > > > > > > > +
> > > > > > > > +.. _PGO: https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization
> > > > > > > > +
> > > > > > > > +
> > > > > > > > +Preparation
> > > > > > > > +===========
> > > > > > > > +
> > > > > > > > +Configure the kernel with:
> > > > > > > > +
> > > > > > > > +.. code-block:: make
> > > > > > > > +
> > > > > > > > + CONFIG_DEBUG_FS=y
> > > > > > > > + CONFIG_PGO_CLANG=y
> > > > > > > > +
> > > > > > > > +Note that kernels compiled with profiling flags will be significantly larger
> > > > > > > > +and run slower.
> > > > > > > > +
> > > > > > > > +Profiling data will only become accessible once debugfs has been mounted:
> > > > > > > > +
> > > > > > > > +.. code-block:: sh
> > > > > > > > +
> > > > > > > > + mount -t debugfs none /sys/kernel/debug
> > > > > > > > +
> > > > > > > > +
> > > > > > > > +Customization
> > > > > > > > +=============
> > > > > > > > +
> > > > > > > > +You can enable or disable profiling for individual file and directories by
> > > > > > > > +adding a line similar to the following to the respective kernel Makefile:
> > > > > > > > +
> > > > > > > > +- For a single file (e.g. main.o)
> > > > > > > > +
> > > > > > > > + .. code-block:: make
> > > > > > > > +
> > > > > > > > + PGO_PROFILE_main.o := y
> > > > > > > > +
> > > > > > > > +- For all files in one directory
> > > > > > > > +
> > > > > > > > + .. code-block:: make
> > > > > > > > +
> > > > > > > > + PGO_PROFILE := y
> > > > > > > > +
> > > > > > > > +To exclude files from being profiled use
> > > > > > > > +
> > > > > > > > + .. code-block:: make
> > > > > > > > +
> > > > > > > > + PGO_PROFILE_main.o := n
> > > > > > > > +
> > > > > > > > +and
> > > > > > > > +
> > > > > > > > + .. code-block:: make
> > > > > > > > +
> > > > > > > > + PGO_PROFILE := n
> > > > > > > > +
> > > > > > > > +Only files which are linked to the main kernel image or are compiled as kernel
> > > > > > > > +modules are supported by this mechanism.
> > > > > > > > +
> > > > > > > > +
> > > > > > > > +Files
> > > > > > > > +=====
> > > > > > > > +
> > > > > > > > +The PGO kernel support creates the following files in debugfs:
> > > > > > > > +
> > > > > > > > +``/sys/kernel/debug/pgo``
> > > > > > > > + Parent directory for all PGO-related files.
> > > > > > > > +
> > > > > > > > +``/sys/kernel/debug/pgo/reset``
> > > > > > > > + Global reset file: resets all coverage data to zero when written to.
> > > > > > > > +
> > > > > > > > +``/sys/kernel/debug/profraw``
> > > > > > > > + The raw PGO data that must be processed with ``llvm_profdata``.
> > > > > > > > +
> > > > > > > > +
> > > > > > > > +Workflow
> > > > > > > > +========
> > > > > > > > +
> > > > > > > > +The PGO kernel can be run on the host or test machines. The data though should
> > > > > > > > +be analyzed with Clang's tools from the same Clang version as the kernel was
> > > > > > > > +compiled. Clang's tolerant of version skew, but it's easier to use the same
> > > > > > > > +Clang version.
> > > > > > > > +
> > > > > > > > +The profiling data is useful for optimizing the kernel, analyzing coverage,
> > > > > > > > +etc. Clang offers tools to perform these tasks.
> > > > > > > > +
> > > > > > > > +Here is an example workflow for profiling an instrumented kernel with PGO and
> > > > > > > > +using the result to optimize the kernel:
> > > > > > > > +
> > > > > > > > +1) Install the kernel on the TEST machine.
> > > > > > > > +
> > > > > > > > +2) Reset the data counters right before running the load tests
> > > > > > > > +
> > > > > > > > + .. code-block:: sh
> > > > > > > > +
> > > > > > > > + $ echo 1 > /sys/kernel/debug/pgo/reset
> > > > > > > > +
> > > > > > >
> > > > > > > I do not get this...
> > > > > > >
> > > > > > > # mount | grep debugfs
> > > > > > > debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
> > > > > > >
> > > > > > > After the load-test...?
> > > > > > >
> > > > > > > echo 0 > /sys/kernel/debug/pgo/reset
> > > > > > >
> > > > > > Writing anything to /sys/kernel/debug/pgo/reset will cause it to reset
> > > > > > the profiling counters. I picked 1 (one) semi-randomly, but it could
> > > > > > be any number, letter, your favorite short story, etc. You don't want
> > > > > > to reset it before collecting the profiling data from your load tests
> > > > > > though.
> > > > > >
> > > > > > > > +3) Run the load tests.
> > > > > > > > +
> > > > > > > > +4) Collect the raw profile data
> > > > > > > > +
> > > > > > > > + .. code-block:: sh
> > > > > > > > +
> > > > > > > > + $ cp -a /sys/kernel/debug/pgo/profraw /tmp/vmlinux.profraw
> > > > > > > > +
> > > > > > >
> > > > > > > This is only 4,9M small and seen from the date 5mins before I run the
> > > > > > > echo-1 line.
> > > > > > >
> > > > > > > # ll /sys/kernel/debug/pgo
> > > > > > > insgesamt 0
> > > > > > > drwxr-xr-x 2 root root 0 16. Jan 17:29 .
> > > > > > > drwx------ 41 root root 0 16. Jan 17:29 ..
> > > > > > > -rw------- 1 root root 0 16. Jan 17:29 profraw
> > > > > > > --w------- 1 root root 0 16. Jan 18:19 reset
> > > > > > >
> > > > > > > # cp -a /sys/kernel/debug/pgo/profraw /tmp/vmlinux.profraw
> > > > > > >
> > > > > > > # ll /tmp/vmlinux.profraw
> > > > > > > -rw------- 1 root root 4,9M 16. Jan 17:29 /tmp/vmlinux.profraw
> > > > > > >
> > > > > > > For me there was no prof-data collected from my defconfig kernel-build.
> > > > > > >
> > > > > > The /sys/kernel/debug/pgo/profraw file is read-only. Nothing writes to
> > > > > > it, not even the kernel. All it does is serialize the profiling
> > > > > > counters from a memory location in the kernel into a format that
> > > > > > LLVM's tools can understand.
> > > > > >
> > > > > > > > +5) (Optional) Download the raw profile data to the HOST machine.
> > > > > > > > +
> > > > > > > > +6) Process the raw profile data
> > > > > > > > +
> > > > > > > > + .. code-block:: sh
> > > > > > > > +
> > > > > > > > + $ llvm-profdata merge --output=vmlinux.profdata vmlinux.profraw
> > > > > > > > +
> > > > > > >
> > > > > > > Is that executed in /path/to/linux/git?
> > > > > > >
> > > > > > The llvm-profdata tool is not in the linux source tree. You need to
> > > > > > grab it from a clang distribution (or built from clang's git repo).
> > > > > >
> > > > > > > > + Note that multiple raw profile data files can be merged during this step.
> > > > > > > > +
> > > > > > > > +7) Rebuild the kernel using the profile data (PGO disabled)
> > > > > > > > +
> > > > > > > > + .. code-block:: sh
> > > > > > > > +
> > > > > > > > + $ make LLVM=1 KCFLAGS=-fprofile-use=vmlinux.profdata ...
> > > > > > >
> > > > > > > How big is vmlinux.profdata (make defconfig)?
> > > > > > >
> > > > > > I don't have numbers for this, but from what you listed here, it's ~5M
> > > > > > in size. The size is proportional to the number of counters
> > > > > > instrumented in the kernel.
> > > > > >
> > > > > > > Do I need to do a full defconfig build or can I stop the build after
> > > > > > > let me say 10mins?
> > > > > > >
> > > > > > You should do a full rebuild. Make sure that PGO is disabled during the rebuild.
> > > > > >
> > > > >
> > > > > Thanks Bill for all the information.
> > > > >
> > > > > And sorry if I am so pedantic.
> > > > >
> > > > > I have installed my Debian system with Legacy-BIOS enabled.
> > > > >
> > > > > When I rebuild with KCFLAGS=-fprofile-use=vmlinux.profdata (LLVM=1 I
> > > > > have as a default) my system hangs on reboot.
> > > > >
> > > > > [ diffconfig ]
> > > > > $ scripts/diffconfig /boot/config-5.11.0-rc3-8-amd64-clang12-pgo
> > > > > /boot/config-5.11.0-rc3-9-amd64-clang12-pgo
> > > > > BUILD_SALT "5.11.0-rc3-8-amd64-clang12-pgo" -> "5.11.0-rc3-9-amd64-clang12-pgo"
> > > > > PGO_CLANG y -> n
> > > > >
> > > > > [ my make line ]
> > > > > $ cat ../start-build_5.11.0-rc3-9-amd64-clang12-pgo.txt
> > > > > dileks 63120 63095 0 06:47 pts/2 00:00:00 /usr/bin/perf_5.10
> > > > > stat make V=1 -j4 HOSTCC=clang HOSTCXX=clang++ HOSTLD=ld.lld CC=clang
> > > > > LD=ld.lld LLVM=1 LLVM_IAS=1 PAHOLE=/opt/pahole/bin/pahole
> > > > > LOCALVERSION=-9-amd64-clang12-pgo KBUILD_VERBOSE=1
> > > > > KBUILD_BUILD_HOST=iniza [email protected]
> > > > > KBUILD_BUILD_TIMESTAMP=2021-01-17 bindeb-pkg
> > > > > KDEB_PKGVERSION=5.11.0~rc3-9~bullseye+dileks1
> > > > > KCFLAGS=-fprofile-use=vmlinux.profdata
> > > > >
> > > > > ( Yes, 06:47 a.m. in the morning :-). )
> > > > >
> > > > > When I boot with the rebuild Linux-kernel I see:
> > > > >
> > > > > Wrong EFI loader signature
> > > > > ...
> > > > > Decompressing
> > > > > Parsing EFI
> > > > > Performing Relocations done.
> > > > > Booting the Kernel.
> > > > >
> > > > > *** SYSTEM HANGS ***
> > > > > ( I waited for approx 1 min )
> > > > >
> > > > > I tried to turn UEFI support ON and OFF.
> > > > > No success.
> > > > >
> > > > > Does Clang-PGO support Legacy-BIOS or is something different wrong?
> > > > >
> > > > > Thanks.
> > > > >
> > > >
> > > > My bootloader is GRUB.
> > > >
> > > > In UEFI-BIOS settings there is no secure-boot disable option.
> > > > Just simple "Use UEFI BIOS" enabled|disabled.
> > > >
> > > > Installed Debian packages:
> > > >
> > > > ii grub-common 2.04-12
> > > > ii grub-pc 2.04-12
> > > > ii grub-pc-bin 2.04-12
> > > > ii grub2-common 2.04-12
> > > >
> > > > I found in the below link to do in grub-shell:
> > > >
> > > > set check_signatures=no
> > > >
> > > > But this is when grub-efi is installed.
> > > >
> > > > - Sedat -
> > > >
> > > > Link: https://unix.stackexchange.com/questions/126286/grub-efi-disable-signature-check
> > >
> > > Forget about that "Wrong EFI bootloader" - I see this with all other
> > > kernels (all boot fine).
> > >
> > > I tried in QEMU with and without KASLR:
> > >
> > > [ run_qemu.sh ]
> > > KPATH=$(pwd)
> > >
> > > APPEND="root=/dev/ram0 console=ttyS0 hung_task_panic=1 earlyprintk=ttyS0,115200"
> > > APPEND="$APPEND nokaslr"
> > >
> > > qemu-system-x86_64 -enable-kvm -M pc -kernel $KPATH/bzImage -initrd
> > > $KPATH/initrd.img -m 512 -net none -serial stdio -append "${APPEND}"
> > > [ /run_qemu.sh ]
> > >
> > > $ ./run_qemu.sh
> > > Probing EDD (edd=off to disable)... ok
> > > Wrong EFI loader signature.
> > > early console in extract_kernel
> > > input_data: 0x000000000289940d
> > > input_len: 0x000000000069804a
> > > output: 0x0000000001000000
> > > output_len: 0x0000000001ef2010
> > > kernel_total_size: 0x0000000001c2c000
> > > needed_size: 0x0000000002000000
> > > trampoline_32bit: 0x000000000009d000
> > >
> > >
> > > KASLR disabled: 'nokaslr' on cmdline.
> > >
> > >
> > > Decompressing Linux... Parsing ELF... No relocation needed... done.
> > > Booting the kernel.
> > >
> > > QEMU run stops, too.
> > >
> >
> > I re-generated my initrd.img with GZIP as compressor (my default is ZSTD).
> >
> > --- /etc/initramfs-tools/initramfs.conf 2021-01-17 12:35:30.823818501 +0100
> > +++ /etc/initramfs-tools/initramfs.conf.zstd 2020-09-21
> > 23:55:43.121735427 +0200
> > @@ -41,7 +41,7 @@ KEYMAP=n
> > # COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz | zstd ]
> > #
> >
> > -COMPRESS=gzip
> > +COMPRESS=zstd
> >
> > #
> > # DEVICE: ...
> >
> > root# KVER="5.11.0-rc3-9-amd64-clang12-pgo" ; update-initramfs -c -k $KVER
> >
> > QEMU boot stops at the same stage.
> >
> > Now, my head is empty...
> >
> > Any comments?
> >
>
> ( Just as a side note I have Nick's DWARF-v5 support enabled. )
>
> There is one EFI related warning in my build-log:
>
> $ grep warning: build-log_5.11.0-rc3-9-amd64-clang12-pgo.txt
> dpkg-architecture: warning: specified GNU system type x86_64-linux-gnu
> does not match CC system type x86_64-pc-linux-gnu, try setting a
> correct CC environment variable
> warning: arch/x86/platform/efi/quirks.c: Function control flow change
> detected (hash mismatch) efi_arch_mem_reserve Hash =
> 391331300655996873 [-Wbackend-plugin]
> warning: arch/x86/platform/efi/efi.c: Function control flow change
> detected (hash mismatch) efi_attr_is_visible Hash = 567185240781730690
> [-Wbackend-plugin]
> arch/x86/crypto/aegis128-aesni-glue.c:265:30: warning: unused variable
> 'simd_alg' [-Wunused-variable]
> warning: lib/crypto/sha256.c: Function control flow change detected
> (hash mismatch) sha256_update Hash = 744640996947387358
> [-Wbackend-plugin]
> warning: arch/x86/boot/compressed/string.c: Function control flow
> change detected (hash mismatch) memcmp Hash = 742261418966908927
> [-Wbackend-plugin]
> warning: arch/x86/boot/compressed/string.c: Function control flow
> change detected (hash mismatch) bcmp Hash = 742261418966908927
> [-Wbackend-plugin]
> warning: arch/x86/boot/compressed/string.c: Function control flow
> change detected (hash mismatch) strcmp Hash = 536873291001348520
> [-Wbackend-plugin]
> warning: arch/x86/boot/compressed/string.c: Function control flow
> change detected (hash mismatch) strnlen Hash = 146835646621254984
> [-Wbackend-plugin]
> warning: arch/x86/boot/compressed/string.c: Function control flow
> change detected (hash mismatch) simple_strtoull Hash =
> 252792765950587360 [-Wbackend-plugin]
> warning: arch/x86/boot/compressed/string.c: Function control flow
> change detected (hash mismatch) strstr Hash = 391331303349076211
> [-Wbackend-plugin]
> warning: arch/x86/boot/compressed/string.c: Function control flow
> change detected (hash mismatch) strchr Hash = 1063705159280644635
> [-Wbackend-plugin]
> warning: arch/x86/boot/compressed/string.c: Function control flow
> change detected (hash mismatch) kstrtoull Hash = 758414239132790022
> [-Wbackend-plugin]
> drivers/infiniband/hw/hfi1/platform.o: warning: objtool: tune_serdes()
> falls through to next function apply_tx_lanes()
>
> Cannot say if this information is helpful.
>

My LLVM/Clang v12 is from <apt.llvm.org>:

clang-12 version 1:12~++20210115111113+45ef053bd709-1~exp1~20210115101809.3724

My kernel-config is attached.

- Sedat -

Attachments:

config-5.11.0-rc3-9-amd64-clang12-pgo (232.65 kB)

2021-01-17 18:06:25

by Sedat Dilek

[permalink] [raw]

Subject: Re: [PATCH v5] pgo: add clang's Profile Guided Optimization infrastructure

On Sun, Jan 17, 2021 at 1:05 PM Sedat Dilek <[email protected]> wrote:
>
> On Sun, Jan 17, 2021 at 12:58 PM Sedat Dilek <[email protected]> wrote:
> >
> > On Sun, Jan 17, 2021 at 12:42 PM Sedat Dilek <[email protected]> wrote:
> > >
> > > On Sun, Jan 17, 2021 at 12:23 PM Sedat Dilek <[email protected]> wrote:
> > > >
> > > > On Sun, Jan 17, 2021 at 11:53 AM Sedat Dilek <[email protected]> wrote:
> > > > >
> > > > > On Sun, Jan 17, 2021 at 11:44 AM Sedat Dilek <[email protected]> wrote:
> > > > > >
> > > > > > On Sat, Jan 16, 2021 at 9:23 PM Bill Wendling <[email protected]> wrote:
> > > > > > >
> > > > > > > On Sat, Jan 16, 2021 at 9:39 AM Sedat Dilek <[email protected]> wrote:
> > > > > > > > On Sat, Jan 16, 2021 at 10:44 AM 'Bill Wendling' via Clang Built Linux
> > > > > > > > <[email protected]> wrote:
> > > > > > > > >
> > > > > > > > > From: Sami Tolvanen <[email protected]>
> > > > > > > > >
> > > > > > > > > Enable the use of clang's Profile-Guided Optimization[1]. To generate a
> > > > > > > > > profile, the kernel is instrumented with PGO counters, a representative
> > > > > > > > > workload is run, and the raw profile data is collected from
> > > > > > > > > /sys/kernel/debug/pgo/profraw.
> > > > > > > > >
> > > > > > > > > The raw profile data must be processed by clang's "llvm-profdata" tool
> > > > > > > > > before it can be used during recompilation:
> > > > > > > > >
> > > > > > > > > $ cp /sys/kernel/debug/pgo/profraw vmlinux.profraw
> > > > > > > > > $ llvm-profdata merge --output=vmlinux.profdata vmlinux.profraw
> > > > > > > > >
> > > > > > > > > Multiple raw profiles may be merged during this step.
> > > > > > > > >
> > > > > > > > > The data can now be used by the compiler:
> > > > > > > > >
> > > > > > > > > $ make LLVM=1 KCFLAGS=-fprofile-use=vmlinux.profdata ...
> > > > > > > > >
> > > > > > > > > This initial submission is restricted to x86, as that's the platform we
> > > > > > > > > know works. This restriction can be lifted once other platforms have
> > > > > > > > > been verified to work with PGO.
> > > > > > > > >
> > > > > > > > > Note that this method of profiling the kernel is clang-native, unlike
> > > > > > > > > the clang support in kernel/gcov.
> > > > > > > > >
> > > > > > > > > [1] https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization
> > > > > > > > >
> > > > > > > > > Signed-off-by: Sami Tolvanen <[email protected]>
> > > > > > > > > Co-developed-by: Bill Wendling <[email protected]>
> > > > > > > > > Signed-off-by: Bill Wendling <[email protected]>
> > > > > > > > > ---
> > > > > > > > > v2: - Added "__llvm_profile_instrument_memop" based on Nathan Chancellor's
> > > > > > > > > testing.
> > > > > > > > > - Corrected documentation, re PGO flags when using LTO, based on Fangrui
> > > > > > > > > Song's comments.
> > > > > > > > > v3: - Added change log section based on Sedat Dilek's comments.
> > > > > > > > > v4: - Remove non-x86 Makfile changes and se "hweight64" instead of using our
> > > > > > > > > own popcount implementation, based on Nick Desaulniers's comment.
> > > > > > > > > v5: - Correct padding calculation, discovered by Nathan Chancellor.
> > > > > > > > > ---
> > > > > > > > > Documentation/dev-tools/index.rst | 1 +
> > > > > > > > > Documentation/dev-tools/pgo.rst | 127 +++++++++
> > > > > > > > > MAINTAINERS | 9 +
> > > > > > > > > Makefile | 3 +
> > > > > > > > > arch/Kconfig | 1 +
> > > > > > > > > arch/x86/Kconfig | 1 +
> > > > > > > > > arch/x86/boot/Makefile | 1 +
> > > > > > > > > arch/x86/boot/compressed/Makefile | 1 +
> > > > > > > > > arch/x86/crypto/Makefile | 2 +
> > > > > > > > > arch/x86/entry/vdso/Makefile | 1 +
> > > > > > > > > arch/x86/kernel/vmlinux.lds.S | 2 +
> > > > > > > > > arch/x86/platform/efi/Makefile | 1 +
> > > > > > > > > arch/x86/purgatory/Makefile | 1 +
> > > > > > > > > arch/x86/realmode/rm/Makefile | 1 +
> > > > > > > > > arch/x86/um/vdso/Makefile | 1 +
> > > > > > > > > drivers/firmware/efi/libstub/Makefile | 1 +
> > > > > > > > > include/asm-generic/vmlinux.lds.h | 44 +++
> > > > > > > > > kernel/Makefile | 1 +
> > > > > > > > > kernel/pgo/Kconfig | 35 +++
> > > > > > > > > kernel/pgo/Makefile | 5 +
> > > > > > > > > kernel/pgo/fs.c | 382 ++++++++++++++++++++++++++
> > > > > > > > > kernel/pgo/instrument.c | 185 +++++++++++++
> > > > > > > > > kernel/pgo/pgo.h | 206 ++++++++++++++
> > > > > > > > > scripts/Makefile.lib | 10 +
> > > > > > > > > 24 files changed, 1022 insertions(+)
> > > > > > > > > create mode 100644 Documentation/dev-tools/pgo.rst
> > > > > > > > > create mode 100644 kernel/pgo/Kconfig
> > > > > > > > > create mode 100644 kernel/pgo/Makefile
> > > > > > > > > create mode 100644 kernel/pgo/fs.c
> > > > > > > > > create mode 100644 kernel/pgo/instrument.c
> > > > > > > > > create mode 100644 kernel/pgo/pgo.h
> > > > > > > > >
> > > > > > > > > diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/index.rst
> > > > > > > > > index f7809c7b1ba9e..8d6418e858062 100644
> > > > > > > > > --- a/Documentation/dev-tools/index.rst
> > > > > > > > > +++ b/Documentation/dev-tools/index.rst
> > > > > > > > > @@ -26,6 +26,7 @@ whole; patches welcome!
> > > > > > > > > kgdb
> > > > > > > > > kselftest
> > > > > > > > > kunit/index
> > > > > > > > > + pgo
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > .. only:: subproject and html
> > > > > > > > > diff --git a/Documentation/dev-tools/pgo.rst b/Documentation/dev-tools/pgo.rst
> > > > > > > > > new file mode 100644
> > > > > > > > > index 0000000000000..b7f11d8405b73
> > > > > > > > > --- /dev/null
> > > > > > > > > +++ b/Documentation/dev-tools/pgo.rst
> > > > > > > > > @@ -0,0 +1,127 @@
> > > > > > > > > +.. SPDX-License-Identifier: GPL-2.0
> > > > > > > > > +
> > > > > > > > > +===============================
> > > > > > > > > +Using PGO with the Linux kernel
> > > > > > > > > +===============================
> > > > > > > > > +
> > > > > > > > > +Clang's profiling kernel support (PGO_) enables profiling of the Linux kernel
> > > > > > > > > +when building with Clang. The profiling data is exported via the ``pgo``
> > > > > > > > > +debugfs directory.
> > > > > > > > > +
> > > > > > > > > +.. _PGO: https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization
> > > > > > > > > +
> > > > > > > > > +
> > > > > > > > > +Preparation
> > > > > > > > > +===========
> > > > > > > > > +
> > > > > > > > > +Configure the kernel with:
> > > > > > > > > +
> > > > > > > > > +.. code-block:: make
> > > > > > > > > +
> > > > > > > > > + CONFIG_DEBUG_FS=y
> > > > > > > > > + CONFIG_PGO_CLANG=y
> > > > > > > > > +
> > > > > > > > > +Note that kernels compiled with profiling flags will be significantly larger
> > > > > > > > > +and run slower.
> > > > > > > > > +
> > > > > > > > > +Profiling data will only become accessible once debugfs has been mounted:
> > > > > > > > > +
> > > > > > > > > +.. code-block:: sh
> > > > > > > > > +
> > > > > > > > > + mount -t debugfs none /sys/kernel/debug
> > > > > > > > > +
> > > > > > > > > +
> > > > > > > > > +Customization
> > > > > > > > > +=============
> > > > > > > > > +
> > > > > > > > > +You can enable or disable profiling for individual file and directories by
> > > > > > > > > +adding a line similar to the following to the respective kernel Makefile:
> > > > > > > > > +
> > > > > > > > > +- For a single file (e.g. main.o)
> > > > > > > > > +
> > > > > > > > > + .. code-block:: make
> > > > > > > > > +
> > > > > > > > > + PGO_PROFILE_main.o := y
> > > > > > > > > +
> > > > > > > > > +- For all files in one directory
> > > > > > > > > +
> > > > > > > > > + .. code-block:: make
> > > > > > > > > +
> > > > > > > > > + PGO_PROFILE := y
> > > > > > > > > +
> > > > > > > > > +To exclude files from being profiled use
> > > > > > > > > +
> > > > > > > > > + .. code-block:: make
> > > > > > > > > +
> > > > > > > > > + PGO_PROFILE_main.o := n
> > > > > > > > > +
> > > > > > > > > +and
> > > > > > > > > +
> > > > > > > > > + .. code-block:: make
> > > > > > > > > +
> > > > > > > > > + PGO_PROFILE := n
> > > > > > > > > +
> > > > > > > > > +Only files which are linked to the main kernel image or are compiled as kernel
> > > > > > > > > +modules are supported by this mechanism.
> > > > > > > > > +
> > > > > > > > > +
> > > > > > > > > +Files
> > > > > > > > > +=====
> > > > > > > > > +
> > > > > > > > > +The PGO kernel support creates the following files in debugfs:
> > > > > > > > > +
> > > > > > > > > +``/sys/kernel/debug/pgo``
> > > > > > > > > + Parent directory for all PGO-related files.
> > > > > > > > > +
> > > > > > > > > +``/sys/kernel/debug/pgo/reset``
> > > > > > > > > + Global reset file: resets all coverage data to zero when written to.
> > > > > > > > > +
> > > > > > > > > +``/sys/kernel/debug/profraw``
> > > > > > > > > + The raw PGO data that must be processed with ``llvm_profdata``.
> > > > > > > > > +
> > > > > > > > > +
> > > > > > > > > +Workflow
> > > > > > > > > +========
> > > > > > > > > +
> > > > > > > > > +The PGO kernel can be run on the host or test machines. The data though should
> > > > > > > > > +be analyzed with Clang's tools from the same Clang version as the kernel was
> > > > > > > > > +compiled. Clang's tolerant of version skew, but it's easier to use the same
> > > > > > > > > +Clang version.
> > > > > > > > > +
> > > > > > > > > +The profiling data is useful for optimizing the kernel, analyzing coverage,
> > > > > > > > > +etc. Clang offers tools to perform these tasks.
> > > > > > > > > +
> > > > > > > > > +Here is an example workflow for profiling an instrumented kernel with PGO and
> > > > > > > > > +using the result to optimize the kernel:
> > > > > > > > > +
> > > > > > > > > +1) Install the kernel on the TEST machine.
> > > > > > > > > +
> > > > > > > > > +2) Reset the data counters right before running the load tests
> > > > > > > > > +
> > > > > > > > > + .. code-block:: sh
> > > > > > > > > +
> > > > > > > > > + $ echo 1 > /sys/kernel/debug/pgo/reset
> > > > > > > > > +
> > > > > > > >
> > > > > > > > I do not get this...
> > > > > > > >
> > > > > > > > # mount | grep debugfs
> > > > > > > > debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
> > > > > > > >
> > > > > > > > After the load-test...?
> > > > > > > >
> > > > > > > > echo 0 > /sys/kernel/debug/pgo/reset
> > > > > > > >
> > > > > > > Writing anything to /sys/kernel/debug/pgo/reset will cause it to reset
> > > > > > > the profiling counters. I picked 1 (one) semi-randomly, but it could
> > > > > > > be any number, letter, your favorite short story, etc. You don't want
> > > > > > > to reset it before collecting the profiling data from your load tests
> > > > > > > though.
> > > > > > >
> > > > > > > > > +3) Run the load tests.
> > > > > > > > > +
> > > > > > > > > +4) Collect the raw profile data
> > > > > > > > > +
> > > > > > > > > + .. code-block:: sh
> > > > > > > > > +
> > > > > > > > > + $ cp -a /sys/kernel/debug/pgo/profraw /tmp/vmlinux.profraw
> > > > > > > > > +
> > > > > > > >
> > > > > > > > This is only 4,9M small and seen from the date 5mins before I run the
> > > > > > > > echo-1 line.
> > > > > > > >
> > > > > > > > # ll /sys/kernel/debug/pgo
> > > > > > > > insgesamt 0
> > > > > > > > drwxr-xr-x 2 root root 0 16. Jan 17:29 .
> > > > > > > > drwx------ 41 root root 0 16. Jan 17:29 ..
> > > > > > > > -rw------- 1 root root 0 16. Jan 17:29 profraw
> > > > > > > > --w------- 1 root root 0 16. Jan 18:19 reset
> > > > > > > >
> > > > > > > > # cp -a /sys/kernel/debug/pgo/profraw /tmp/vmlinux.profraw
> > > > > > > >
> > > > > > > > # ll /tmp/vmlinux.profraw
> > > > > > > > -rw------- 1 root root 4,9M 16. Jan 17:29 /tmp/vmlinux.profraw
> > > > > > > >
> > > > > > > > For me there was no prof-data collected from my defconfig kernel-build.
> > > > > > > >
> > > > > > > The /sys/kernel/debug/pgo/profraw file is read-only. Nothing writes to
> > > > > > > it, not even the kernel. All it does is serialize the profiling
> > > > > > > counters from a memory location in the kernel into a format that
> > > > > > > LLVM's tools can understand.
> > > > > > >
> > > > > > > > > +5) (Optional) Download the raw profile data to the HOST machine.
> > > > > > > > > +
> > > > > > > > > +6) Process the raw profile data
> > > > > > > > > +
> > > > > > > > > + .. code-block:: sh
> > > > > > > > > +
> > > > > > > > > + $ llvm-profdata merge --output=vmlinux.profdata vmlinux.profraw
> > > > > > > > > +
> > > > > > > >
> > > > > > > > Is that executed in /path/to/linux/git?
> > > > > > > >
> > > > > > > The llvm-profdata tool is not in the linux source tree. You need to
> > > > > > > grab it from a clang distribution (or built from clang's git repo).
> > > > > > >
> > > > > > > > > + Note that multiple raw profile data files can be merged during this step.
> > > > > > > > > +
> > > > > > > > > +7) Rebuild the kernel using the profile data (PGO disabled)
> > > > > > > > > +
> > > > > > > > > + .. code-block:: sh
> > > > > > > > > +
> > > > > > > > > + $ make LLVM=1 KCFLAGS=-fprofile-use=vmlinux.profdata ...
> > > > > > > >
> > > > > > > > How big is vmlinux.profdata (make defconfig)?
> > > > > > > >
> > > > > > > I don't have numbers for this, but from what you listed here, it's ~5M
> > > > > > > in size. The size is proportional to the number of counters
> > > > > > > instrumented in the kernel.
> > > > > > >
> > > > > > > > Do I need to do a full defconfig build or can I stop the build after
> > > > > > > > let me say 10mins?
> > > > > > > >
> > > > > > > You should do a full rebuild. Make sure that PGO is disabled during the rebuild.
> > > > > > >
> > > > > >
> > > > > > Thanks Bill for all the information.
> > > > > >
> > > > > > And sorry if I am so pedantic.
> > > > > >
> > > > > > I have installed my Debian system with Legacy-BIOS enabled.
> > > > > >
> > > > > > When I rebuild with KCFLAGS=-fprofile-use=vmlinux.profdata (LLVM=1 I
> > > > > > have as a default) my system hangs on reboot.
> > > > > >
> > > > > > [ diffconfig ]
> > > > > > $ scripts/diffconfig /boot/config-5.11.0-rc3-8-amd64-clang12-pgo
> > > > > > /boot/config-5.11.0-rc3-9-amd64-clang12-pgo
> > > > > > BUILD_SALT "5.11.0-rc3-8-amd64-clang12-pgo" -> "5.11.0-rc3-9-amd64-clang12-pgo"
> > > > > > PGO_CLANG y -> n
> > > > > >
> > > > > > [ my make line ]
> > > > > > $ cat ../start-build_5.11.0-rc3-9-amd64-clang12-pgo.txt
> > > > > > dileks 63120 63095 0 06:47 pts/2 00:00:00 /usr/bin/perf_5.10
> > > > > > stat make V=1 -j4 HOSTCC=clang HOSTCXX=clang++ HOSTLD=ld.lld CC=clang
> > > > > > LD=ld.lld LLVM=1 LLVM_IAS=1 PAHOLE=/opt/pahole/bin/pahole
> > > > > > LOCALVERSION=-9-amd64-clang12-pgo KBUILD_VERBOSE=1
> > > > > > KBUILD_BUILD_HOST=iniza [email protected]
> > > > > > KBUILD_BUILD_TIMESTAMP=2021-01-17 bindeb-pkg
> > > > > > KDEB_PKGVERSION=5.11.0~rc3-9~bullseye+dileks1
> > > > > > KCFLAGS=-fprofile-use=vmlinux.profdata
> > > > > >
> > > > > > ( Yes, 06:47 a.m. in the morning :-). )
> > > > > >
> > > > > > When I boot with the rebuild Linux-kernel I see:
> > > > > >
> > > > > > Wrong EFI loader signature
> > > > > > ...
> > > > > > Decompressing
> > > > > > Parsing EFI
> > > > > > Performing Relocations done.
> > > > > > Booting the Kernel.
> > > > > >
> > > > > > *** SYSTEM HANGS ***
> > > > > > ( I waited for approx 1 min )
> > > > > >
> > > > > > I tried to turn UEFI support ON and OFF.
> > > > > > No success.
> > > > > >
> > > > > > Does Clang-PGO support Legacy-BIOS or is something different wrong?
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > >
> > > > > My bootloader is GRUB.
> > > > >
> > > > > In UEFI-BIOS settings there is no secure-boot disable option.
> > > > > Just simple "Use UEFI BIOS" enabled|disabled.
> > > > >
> > > > > Installed Debian packages:
> > > > >
> > > > > ii grub-common 2.04-12
> > > > > ii grub-pc 2.04-12
> > > > > ii grub-pc-bin 2.04-12
> > > > > ii grub2-common 2.04-12
> > > > >
> > > > > I found in the below link to do in grub-shell:
> > > > >
> > > > > set check_signatures=no
> > > > >
> > > > > But this is when grub-efi is installed.
> > > > >
> > > > > - Sedat -
> > > > >
> > > > > Link: https://unix.stackexchange.com/questions/126286/grub-efi-disable-signature-check
> > > >
> > > > Forget about that "Wrong EFI bootloader" - I see this with all other
> > > > kernels (all boot fine).
> > > >
> > > > I tried in QEMU with and without KASLR:
> > > >
> > > > [ run_qemu.sh ]
> > > > KPATH=$(pwd)
> > > >
> > > > APPEND="root=/dev/ram0 console=ttyS0 hung_task_panic=1 earlyprintk=ttyS0,115200"
> > > > APPEND="$APPEND nokaslr"
> > > >
> > > > qemu-system-x86_64 -enable-kvm -M pc -kernel $KPATH/bzImage -initrd
> > > > $KPATH/initrd.img -m 512 -net none -serial stdio -append "${APPEND}"
> > > > [ /run_qemu.sh ]
> > > >
> > > > $ ./run_qemu.sh
> > > > Probing EDD (edd=off to disable)... ok
> > > > Wrong EFI loader signature.
> > > > early console in extract_kernel
> > > > input_data: 0x000000000289940d
> > > > input_len: 0x000000000069804a
> > > > output: 0x0000000001000000
> > > > output_len: 0x0000000001ef2010
> > > > kernel_total_size: 0x0000000001c2c000
> > > > needed_size: 0x0000000002000000
> > > > trampoline_32bit: 0x000000000009d000
> > > >
> > > >
> > > > KASLR disabled: 'nokaslr' on cmdline.
> > > >
> > > >
> > > > Decompressing Linux... Parsing ELF... No relocation needed... done.
> > > > Booting the kernel.
> > > >
> > > > QEMU run stops, too.
> > > >
> > >
> > > I re-generated my initrd.img with GZIP as compressor (my default is ZSTD).
> > >
> > > --- /etc/initramfs-tools/initramfs.conf 2021-01-17 12:35:30.823818501 +0100
> > > +++ /etc/initramfs-tools/initramfs.conf.zstd 2020-09-21
> > > 23:55:43.121735427 +0200
> > > @@ -41,7 +41,7 @@ KEYMAP=n
> > > # COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz | zstd ]
> > > #
> > >
> > > -COMPRESS=gzip
> > > +COMPRESS=zstd
> > >
> > > #
> > > # DEVICE: ...
> > >
> > > root# KVER="5.11.0-rc3-9-amd64-clang12-pgo" ; update-initramfs -c -k $KVER
> > >
> > > QEMU boot stops at the same stage.
> > >
> > > Now, my head is empty...
> > >
> > > Any comments?
> > >
> >
> > ( Just as a side note I have Nick's DWARF-v5 support enabled. )
> >
> > There is one EFI related warning in my build-log:
> >
> > $ grep warning: build-log_5.11.0-rc3-9-amd64-clang12-pgo.txt
> > dpkg-architecture: warning: specified GNU system type x86_64-linux-gnu
> > does not match CC system type x86_64-pc-linux-gnu, try setting a
> > correct CC environment variable
> > warning: arch/x86/platform/efi/quirks.c: Function control flow change
> > detected (hash mismatch) efi_arch_mem_reserve Hash =
> > 391331300655996873 [-Wbackend-plugin]
> > warning: arch/x86/platform/efi/efi.c: Function control flow change
> > detected (hash mismatch) efi_attr_is_visible Hash = 567185240781730690
> > [-Wbackend-plugin]
> > arch/x86/crypto/aegis128-aesni-glue.c:265:30: warning: unused variable
> > 'simd_alg' [-Wunused-variable]
> > warning: lib/crypto/sha256.c: Function control flow change detected
> > (hash mismatch) sha256_update Hash = 744640996947387358
> > [-Wbackend-plugin]
> > warning: arch/x86/boot/compressed/string.c: Function control flow
> > change detected (hash mismatch) memcmp Hash = 742261418966908927
> > [-Wbackend-plugin]
> > warning: arch/x86/boot/compressed/string.c: Function control flow
> > change detected (hash mismatch) bcmp Hash = 742261418966908927
> > [-Wbackend-plugin]
> > warning: arch/x86/boot/compressed/string.c: Function control flow
> > change detected (hash mismatch) strcmp Hash = 536873291001348520
> > [-Wbackend-plugin]
> > warning: arch/x86/boot/compressed/string.c: Function control flow
> > change detected (hash mismatch) strnlen Hash = 146835646621254984
> > [-Wbackend-plugin]
> > warning: arch/x86/boot/compressed/string.c: Function control flow
> > change detected (hash mismatch) simple_strtoull Hash =
> > 252792765950587360 [-Wbackend-plugin]
> > warning: arch/x86/boot/compressed/string.c: Function control flow
> > change detected (hash mismatch) strstr Hash = 391331303349076211
> > [-Wbackend-plugin]
> > warning: arch/x86/boot/compressed/string.c: Function control flow
> > change detected (hash mismatch) strchr Hash = 1063705159280644635
> > [-Wbackend-plugin]
> > warning: arch/x86/boot/compressed/string.c: Function control flow
> > change detected (hash mismatch) kstrtoull Hash = 758414239132790022
> > [-Wbackend-plugin]
> > drivers/infiniband/hw/hfi1/platform.o: warning: objtool: tune_serdes()
> > falls through to next function apply_tx_lanes()
> >
> > Cannot say if this information is helpful.
> >
>
> My LLVM/Clang v12 is from <apt.llvm.org>:
>
> clang-12 version 1:12~++20210115111113+45ef053bd709-1~exp1~20210115101809.3724
>
> My kernel-config is attached.
>

I dropped "LLVM_IAS=1" from my make line and did for my next build:

$ scripts/diffconfig /boot/config-5.11.0-rc3-8-amd64-clang12-pgo .config
BUILD_SALT "5.11.0-rc3-8-amd64-clang12-pgo" -> "5.11.0-rc3-10-amd64-clang12-pgo"
DEBUG_INFO_DWARF2 n -> y
DEBUG_INFO_DWARF5 y -> n
PGO_CLANG y -> n

Means dropped DWARF5 support.

- Sedat -

2021-01-17 21:12:15

by Bill Wendling

[permalink] [raw]

Subject: Re: [PATCH v5] pgo: add clang's Profile Guided Optimization infrastructure

On Sun, Jan 17, 2021 at 9:42 AM Sedat Dilek <[email protected]> wrote:
>
> On Sun, Jan 17, 2021 at 1:05 PM Sedat Dilek <[email protected]> wrote:
> >
> > On Sun, Jan 17, 2021 at 12:58 PM Sedat Dilek <[email protected]> wrote:
> > >
> > > On Sun, Jan 17, 2021 at 12:42 PM Sedat Dilek <[email protected]> wrote:
> > > >
> > > > On Sun, Jan 17, 2021 at 12:23 PM Sedat Dilek <[email protected]> wrote:
> > > > >
> > > > > On Sun, Jan 17, 2021 at 11:53 AM Sedat Dilek <[email protected]> wrote:
> > > > > >
> > > > > > On Sun, Jan 17, 2021 at 11:44 AM Sedat Dilek <[email protected]> wrote:
> > > > > > >
> > > > > > > On Sat, Jan 16, 2021 at 9:23 PM Bill Wendling <[email protected]> wrote:
> > > > > > > >
> > > > > > > > On Sat, Jan 16, 2021 at 9:39 AM Sedat Dilek <[email protected]> wrote:
> > > > > > > > > On Sat, Jan 16, 2021 at 10:44 AM 'Bill Wendling' via Clang Built Linux
> > > > > > > > > <[email protected]> wrote:
> > > > > > > > > >
> > > > > > > > > > From: Sami Tolvanen <[email protected]>
> > > > > > > > > >
> > > > > > > > > > Enable the use of clang's Profile-Guided Optimization[1]. To generate a
> > > > > > > > > > profile, the kernel is instrumented with PGO counters, a representative
> > > > > > > > > > workload is run, and the raw profile data is collected from
> > > > > > > > > > /sys/kernel/debug/pgo/profraw.
> > > > > > > > > >
> > > > > > > > > > The raw profile data must be processed by clang's "llvm-profdata" tool
> > > > > > > > > > before it can be used during recompilation:
> > > > > > > > > >
> > > > > > > > > > $ cp /sys/kernel/debug/pgo/profraw vmlinux.profraw
> > > > > > > > > > $ llvm-profdata merge --output=vmlinux.profdata vmlinux.profraw
> > > > > > > > > >
> > > > > > > > > > Multiple raw profiles may be merged during this step.
> > > > > > > > > >
> > > > > > > > > > The data can now be used by the compiler:
> > > > > > > > > >
> > > > > > > > > > $ make LLVM=1 KCFLAGS=-fprofile-use=vmlinux.profdata ...
> > > > > > > > > >
> > > > > > > > > > This initial submission is restricted to x86, as that's the platform we
> > > > > > > > > > know works. This restriction can be lifted once other platforms have
> > > > > > > > > > been verified to work with PGO.
> > > > > > > > > >
> > > > > > > > > > Note that this method of profiling the kernel is clang-native, unlike
> > > > > > > > > > the clang support in kernel/gcov.
> > > > > > > > > >
> > > > > > > > > > [1] https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Sami Tolvanen <[email protected]>
> > > > > > > > > > Co-developed-by: Bill Wendling <[email protected]>
> > > > > > > > > > Signed-off-by: Bill Wendling <[email protected]>
> > > > > > > > > > ---
> > > > > > > > > > v2: - Added "__llvm_profile_instrument_memop" based on Nathan Chancellor's
> > > > > > > > > > testing.
> > > > > > > > > > - Corrected documentation, re PGO flags when using LTO, based on Fangrui
> > > > > > > > > > Song's comments.
> > > > > > > > > > v3: - Added change log section based on Sedat Dilek's comments.
> > > > > > > > > > v4: - Remove non-x86 Makfile changes and se "hweight64" instead of using our
> > > > > > > > > > own popcount implementation, based on Nick Desaulniers's comment.
> > > > > > > > > > v5: - Correct padding calculation, discovered by Nathan Chancellor.
> > > > > > > > > > ---
> > > > > > > > > > Documentation/dev-tools/index.rst | 1 +
> > > > > > > > > > Documentation/dev-tools/pgo.rst | 127 +++++++++
> > > > > > > > > > MAINTAINERS | 9 +
> > > > > > > > > > Makefile | 3 +
> > > > > > > > > > arch/Kconfig | 1 +
> > > > > > > > > > arch/x86/Kconfig | 1 +
> > > > > > > > > > arch/x86/boot/Makefile | 1 +
> > > > > > > > > > arch/x86/boot/compressed/Makefile | 1 +
> > > > > > > > > > arch/x86/crypto/Makefile | 2 +
> > > > > > > > > > arch/x86/entry/vdso/Makefile | 1 +
> > > > > > > > > > arch/x86/kernel/vmlinux.lds.S | 2 +
> > > > > > > > > > arch/x86/platform/efi/Makefile | 1 +
> > > > > > > > > > arch/x86/purgatory/Makefile | 1 +
> > > > > > > > > > arch/x86/realmode/rm/Makefile | 1 +
> > > > > > > > > > arch/x86/um/vdso/Makefile | 1 +
> > > > > > > > > > drivers/firmware/efi/libstub/Makefile | 1 +
> > > > > > > > > > include/asm-generic/vmlinux.lds.h | 44 +++
> > > > > > > > > > kernel/Makefile | 1 +
> > > > > > > > > > kernel/pgo/Kconfig | 35 +++
> > > > > > > > > > kernel/pgo/Makefile | 5 +
> > > > > > > > > > kernel/pgo/fs.c | 382 ++++++++++++++++++++++++++
> > > > > > > > > > kernel/pgo/instrument.c | 185 +++++++++++++
> > > > > > > > > > kernel/pgo/pgo.h | 206 ++++++++++++++
> > > > > > > > > > scripts/Makefile.lib | 10 +
> > > > > > > > > > 24 files changed, 1022 insertions(+)
> > > > > > > > > > create mode 100644 Documentation/dev-tools/pgo.rst
> > > > > > > > > > create mode 100644 kernel/pgo/Kconfig
> > > > > > > > > > create mode 100644 kernel/pgo/Makefile
> > > > > > > > > > create mode 100644 kernel/pgo/fs.c
> > > > > > > > > > create mode 100644 kernel/pgo/instrument.c
> > > > > > > > > > create mode 100644 kernel/pgo/pgo.h
> > > > > > > > > >
> > > > > > > > > > diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/index.rst
> > > > > > > > > > index f7809c7b1ba9e..8d6418e858062 100644
> > > > > > > > > > --- a/Documentation/dev-tools/index.rst
> > > > > > > > > > +++ b/Documentation/dev-tools/index.rst
> > > > > > > > > > @@ -26,6 +26,7 @@ whole; patches welcome!
> > > > > > > > > > kgdb
> > > > > > > > > > kselftest
> > > > > > > > > > kunit/index
> > > > > > > > > > + pgo
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > .. only:: subproject and html
> > > > > > > > > > diff --git a/Documentation/dev-tools/pgo.rst b/Documentation/dev-tools/pgo.rst
> > > > > > > > > > new file mode 100644
> > > > > > > > > > index 0000000000000..b7f11d8405b73
> > > > > > > > > > --- /dev/null
> > > > > > > > > > +++ b/Documentation/dev-tools/pgo.rst
> > > > > > > > > > @@ -0,0 +1,127 @@
> > > > > > > > > > +.. SPDX-License-Identifier: GPL-2.0
> > > > > > > > > > +
> > > > > > > > > > +===============================
> > > > > > > > > > +Using PGO with the Linux kernel
> > > > > > > > > > +===============================
> > > > > > > > > > +
> > > > > > > > > > +Clang's profiling kernel support (PGO_) enables profiling of the Linux kernel
> > > > > > > > > > +when building with Clang. The profiling data is exported via the ``pgo``
> > > > > > > > > > +debugfs directory.
> > > > > > > > > > +
> > > > > > > > > > +.. _PGO: https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization
> > > > > > > > > > +
> > > > > > > > > > +
> > > > > > > > > > +Preparation
> > > > > > > > > > +===========
> > > > > > > > > > +
> > > > > > > > > > +Configure the kernel with:
> > > > > > > > > > +
> > > > > > > > > > +.. code-block:: make
> > > > > > > > > > +
> > > > > > > > > > + CONFIG_DEBUG_FS=y
> > > > > > > > > > + CONFIG_PGO_CLANG=y
> > > > > > > > > > +
> > > > > > > > > > +Note that kernels compiled with profiling flags will be significantly larger
> > > > > > > > > > +and run slower.
> > > > > > > > > > +
> > > > > > > > > > +Profiling data will only become accessible once debugfs has been mounted:
> > > > > > > > > > +
> > > > > > > > > > +.. code-block:: sh
> > > > > > > > > > +
> > > > > > > > > > + mount -t debugfs none /sys/kernel/debug
> > > > > > > > > > +
> > > > > > > > > > +
> > > > > > > > > > +Customization
> > > > > > > > > > +=============
> > > > > > > > > > +
> > > > > > > > > > +You can enable or disable profiling for individual file and directories by
> > > > > > > > > > +adding a line similar to the following to the respective kernel Makefile:
> > > > > > > > > > +
> > > > > > > > > > +- For a single file (e.g. main.o)
> > > > > > > > > > +
> > > > > > > > > > + .. code-block:: make
> > > > > > > > > > +
> > > > > > > > > > + PGO_PROFILE_main.o := y
> > > > > > > > > > +
> > > > > > > > > > +- For all files in one directory
> > > > > > > > > > +
> > > > > > > > > > + .. code-block:: make
> > > > > > > > > > +
> > > > > > > > > > + PGO_PROFILE := y
> > > > > > > > > > +
> > > > > > > > > > +To exclude files from being profiled use
> > > > > > > > > > +
> > > > > > > > > > + .. code-block:: make
> > > > > > > > > > +
> > > > > > > > > > + PGO_PROFILE_main.o := n
> > > > > > > > > > +
> > > > > > > > > > +and
> > > > > > > > > > +
> > > > > > > > > > + .. code-block:: make
> > > > > > > > > > +
> > > > > > > > > > + PGO_PROFILE := n
> > > > > > > > > > +
> > > > > > > > > > +Only files which are linked to the main kernel image or are compiled as kernel
> > > > > > > > > > +modules are supported by this mechanism.
> > > > > > > > > > +
> > > > > > > > > > +
> > > > > > > > > > +Files
> > > > > > > > > > +=====
> > > > > > > > > > +
> > > > > > > > > > +The PGO kernel support creates the following files in debugfs:
> > > > > > > > > > +
> > > > > > > > > > +``/sys/kernel/debug/pgo``
> > > > > > > > > > + Parent directory for all PGO-related files.
> > > > > > > > > > +
> > > > > > > > > > +``/sys/kernel/debug/pgo/reset``
> > > > > > > > > > + Global reset file: resets all coverage data to zero when written to.
> > > > > > > > > > +
> > > > > > > > > > +``/sys/kernel/debug/profraw``
> > > > > > > > > > + The raw PGO data that must be processed with ``llvm_profdata``.
> > > > > > > > > > +
> > > > > > > > > > +
> > > > > > > > > > +Workflow
> > > > > > > > > > +========
> > > > > > > > > > +
> > > > > > > > > > +The PGO kernel can be run on the host or test machines. The data though should
> > > > > > > > > > +be analyzed with Clang's tools from the same Clang version as the kernel was
> > > > > > > > > > +compiled. Clang's tolerant of version skew, but it's easier to use the same
> > > > > > > > > > +Clang version.
> > > > > > > > > > +
> > > > > > > > > > +The profiling data is useful for optimizing the kernel, analyzing coverage,
> > > > > > > > > > +etc. Clang offers tools to perform these tasks.
> > > > > > > > > > +
> > > > > > > > > > +Here is an example workflow for profiling an instrumented kernel with PGO and
> > > > > > > > > > +using the result to optimize the kernel:
> > > > > > > > > > +
> > > > > > > > > > +1) Install the kernel on the TEST machine.
> > > > > > > > > > +
> > > > > > > > > > +2) Reset the data counters right before running the load tests
> > > > > > > > > > +
> > > > > > > > > > + .. code-block:: sh
> > > > > > > > > > +
> > > > > > > > > > + $ echo 1 > /sys/kernel/debug/pgo/reset
> > > > > > > > > > +
> > > > > > > > >
> > > > > > > > > I do not get this...
> > > > > > > > >
> > > > > > > > > # mount | grep debugfs
> > > > > > > > > debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
> > > > > > > > >
> > > > > > > > > After the load-test...?
> > > > > > > > >
> > > > > > > > > echo 0 > /sys/kernel/debug/pgo/reset
> > > > > > > > >
> > > > > > > > Writing anything to /sys/kernel/debug/pgo/reset will cause it to reset
> > > > > > > > the profiling counters. I picked 1 (one) semi-randomly, but it could
> > > > > > > > be any number, letter, your favorite short story, etc. You don't want
> > > > > > > > to reset it before collecting the profiling data from your load tests
> > > > > > > > though.
> > > > > > > >
> > > > > > > > > > +3) Run the load tests.
> > > > > > > > > > +
> > > > > > > > > > +4) Collect the raw profile data
> > > > > > > > > > +
> > > > > > > > > > + .. code-block:: sh
> > > > > > > > > > +
> > > > > > > > > > + $ cp -a /sys/kernel/debug/pgo/profraw /tmp/vmlinux.profraw
> > > > > > > > > > +
> > > > > > > > >
> > > > > > > > > This is only 4,9M small and seen from the date 5mins before I run the
> > > > > > > > > echo-1 line.
> > > > > > > > >
> > > > > > > > > # ll /sys/kernel/debug/pgo
> > > > > > > > > insgesamt 0
> > > > > > > > > drwxr-xr-x 2 root root 0 16. Jan 17:29 .
> > > > > > > > > drwx------ 41 root root 0 16. Jan 17:29 ..
> > > > > > > > > -rw------- 1 root root 0 16. Jan 17:29 profraw
> > > > > > > > > --w------- 1 root root 0 16. Jan 18:19 reset
> > > > > > > > >
> > > > > > > > > # cp -a /sys/kernel/debug/pgo/profraw /tmp/vmlinux.profraw
> > > > > > > > >
> > > > > > > > > # ll /tmp/vmlinux.profraw
> > > > > > > > > -rw------- 1 root root 4,9M 16. Jan 17:29 /tmp/vmlinux.profraw
> > > > > > > > >
> > > > > > > > > For me there was no prof-data collected from my defconfig kernel-build.
> > > > > > > > >
> > > > > > > > The /sys/kernel/debug/pgo/profraw file is read-only. Nothing writes to
> > > > > > > > it, not even the kernel. All it does is serialize the profiling
> > > > > > > > counters from a memory location in the kernel into a format that
> > > > > > > > LLVM's tools can understand.
> > > > > > > >
> > > > > > > > > > +5) (Optional) Download the raw profile data to the HOST machine.
> > > > > > > > > > +
> > > > > > > > > > +6) Process the raw profile data
> > > > > > > > > > +
> > > > > > > > > > + .. code-block:: sh
> > > > > > > > > > +
> > > > > > > > > > + $ llvm-profdata merge --output=vmlinux.profdata vmlinux.profraw
> > > > > > > > > > +
> > > > > > > > >
> > > > > > > > > Is that executed in /path/to/linux/git?
> > > > > > > > >
> > > > > > > > The llvm-profdata tool is not in the linux source tree. You need to
> > > > > > > > grab it from a clang distribution (or built from clang's git repo).
> > > > > > > >
> > > > > > > > > > + Note that multiple raw profile data files can be merged during this step.
> > > > > > > > > > +
> > > > > > > > > > +7) Rebuild the kernel using the profile data (PGO disabled)
> > > > > > > > > > +
> > > > > > > > > > + .. code-block:: sh
> > > > > > > > > > +
> > > > > > > > > > + $ make LLVM=1 KCFLAGS=-fprofile-use=vmlinux.profdata ...
> > > > > > > > >
> > > > > > > > > How big is vmlinux.profdata (make defconfig)?
> > > > > > > > >
> > > > > > > > I don't have numbers for this, but from what you listed here, it's ~5M
> > > > > > > > in size. The size is proportional to the number of counters
> > > > > > > > instrumented in the kernel.
> > > > > > > >
> > > > > > > > > Do I need to do a full defconfig build or can I stop the build after
> > > > > > > > > let me say 10mins?
> > > > > > > > >
> > > > > > > > You should do a full rebuild. Make sure that PGO is disabled during the rebuild.
> > > > > > > >
> > > > > > >
> > > > > > > Thanks Bill for all the information.
> > > > > > >
> > > > > > > And sorry if I am so pedantic.
> > > > > > >
> > > > > > > I have installed my Debian system with Legacy-BIOS enabled.
> > > > > > >
> > > > > > > When I rebuild with KCFLAGS=-fprofile-use=vmlinux.profdata (LLVM=1 I
> > > > > > > have as a default) my system hangs on reboot.
> > > > > > >
> > > > > > > [ diffconfig ]
> > > > > > > $ scripts/diffconfig /boot/config-5.11.0-rc3-8-amd64-clang12-pgo
> > > > > > > /boot/config-5.11.0-rc3-9-amd64-clang12-pgo
> > > > > > > BUILD_SALT "5.11.0-rc3-8-amd64-clang12-pgo" -> "5.11.0-rc3-9-amd64-clang12-pgo"
> > > > > > > PGO_CLANG y -> n
> > > > > > >
> > > > > > > [ my make line ]
> > > > > > > $ cat ../start-build_5.11.0-rc3-9-amd64-clang12-pgo.txt
> > > > > > > dileks 63120 63095 0 06:47 pts/2 00:00:00 /usr/bin/perf_5.10
> > > > > > > stat make V=1 -j4 HOSTCC=clang HOSTCXX=clang++ HOSTLD=ld.lld CC=clang
> > > > > > > LD=ld.lld LLVM=1 LLVM_IAS=1 PAHOLE=/opt/pahole/bin/pahole
> > > > > > > LOCALVERSION=-9-amd64-clang12-pgo KBUILD_VERBOSE=1
> > > > > > > KBUILD_BUILD_HOST=iniza [email protected]
> > > > > > > KBUILD_BUILD_TIMESTAMP=2021-01-17 bindeb-pkg
> > > > > > > KDEB_PKGVERSION=5.11.0~rc3-9~bullseye+dileks1
> > > > > > > KCFLAGS=-fprofile-use=vmlinux.profdata
> > > > > > >
> > > > > > > ( Yes, 06:47 a.m. in the morning :-). )
> > > > > > >
> > > > > > > When I boot with the rebuild Linux-kernel I see:
> > > > > > >
> > > > > > > Wrong EFI loader signature
> > > > > > > ...
> > > > > > > Decompressing
> > > > > > > Parsing EFI
> > > > > > > Performing Relocations done.
> > > > > > > Booting the Kernel.
> > > > > > >
> > > > > > > *** SYSTEM HANGS ***
> > > > > > > ( I waited for approx 1 min )
> > > > > > >
> > > > > > > I tried to turn UEFI support ON and OFF.
> > > > > > > No success.
> > > > > > >
> > > > > > > Does Clang-PGO support Legacy-BIOS or is something different wrong?
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > >
> > > > > > My bootloader is GRUB.
> > > > > >
> > > > > > In UEFI-BIOS settings there is no secure-boot disable option.
> > > > > > Just simple "Use UEFI BIOS" enabled|disabled.
> > > > > >
> > > > > > Installed Debian packages:
> > > > > >
> > > > > > ii grub-common 2.04-12
> > > > > > ii grub-pc 2.04-12
> > > > > > ii grub-pc-bin 2.04-12
> > > > > > ii grub2-common 2.04-12
> > > > > >
> > > > > > I found in the below link to do in grub-shell:
> > > > > >
> > > > > > set check_signatures=no
> > > > > >
> > > > > > But this is when grub-efi is installed.
> > > > > >
> > > > > > - Sedat -
> > > > > >
> > > > > > Link: https://unix.stackexchange.com/questions/126286/grub-efi-disable-signature-check
> > > > >
> > > > > Forget about that "Wrong EFI bootloader" - I see this with all other
> > > > > kernels (all boot fine).
> > > > >
> > > > > I tried in QEMU with and without KASLR:
> > > > >
> > > > > [ run_qemu.sh ]
> > > > > KPATH=$(pwd)
> > > > >
> > > > > APPEND="root=/dev/ram0 console=ttyS0 hung_task_panic=1 earlyprintk=ttyS0,115200"
> > > > > APPEND="$APPEND nokaslr"
> > > > >
> > > > > qemu-system-x86_64 -enable-kvm -M pc -kernel $KPATH/bzImage -initrd
> > > > > $KPATH/initrd.img -m 512 -net none -serial stdio -append "${APPEND}"
> > > > > [ /run_qemu.sh ]
> > > > >
> > > > > $ ./run_qemu.sh
> > > > > Probing EDD (edd=off to disable)... ok
> > > > > Wrong EFI loader signature.
> > > > > early console in extract_kernel
> > > > > input_data: 0x000000000289940d
> > > > > input_len: 0x000000000069804a
> > > > > output: 0x0000000001000000
> > > > > output_len: 0x0000000001ef2010
> > > > > kernel_total_size: 0x0000000001c2c000
> > > > > needed_size: 0x0000000002000000
> > > > > trampoline_32bit: 0x000000000009d000
> > > > >
> > > > >
> > > > > KASLR disabled: 'nokaslr' on cmdline.
> > > > >
> > > > >
> > > > > Decompressing Linux... Parsing ELF... No relocation needed... done.
> > > > > Booting the kernel.
> > > > >
> > > > > QEMU run stops, too.
> > > > >
> > > >
> > > > I re-generated my initrd.img with GZIP as compressor (my default is ZSTD).
> > > >
> > > > --- /etc/initramfs-tools/initramfs.conf 2021-01-17 12:35:30.823818501 +0100
> > > > +++ /etc/initramfs-tools/initramfs.conf.zstd 2020-09-21
> > > > 23:55:43.121735427 +0200
> > > > @@ -41,7 +41,7 @@ KEYMAP=n
> > > > # COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz | zstd ]
> > > > #
> > > >
> > > > -COMPRESS=gzip
> > > > +COMPRESS=zstd
> > > >
> > > > #
> > > > # DEVICE: ...
> > > >
> > > > root# KVER="5.11.0-rc3-9-amd64-clang12-pgo" ; update-initramfs -c -k $KVER
> > > >
> > > > QEMU boot stops at the same stage.
> > > >
> > > > Now, my head is empty...
> > > >
> > > > Any comments?
> > > >
> > >
> > > ( Just as a side note I have Nick's DWARF-v5 support enabled. )
> > >
> > > There is one EFI related warning in my build-log:
> > >
> > > $ grep warning: build-log_5.11.0-rc3-9-amd64-clang12-pgo.txt
> > > dpkg-architecture: warning: specified GNU system type x86_64-linux-gnu
> > > does not match CC system type x86_64-pc-linux-gnu, try setting a
> > > correct CC environment variable
> > > warning: arch/x86/platform/efi/quirks.c: Function control flow change
> > > detected (hash mismatch) efi_arch_mem_reserve Hash =
> > > 391331300655996873 [-Wbackend-plugin]
> > > warning: arch/x86/platform/efi/efi.c: Function control flow change
> > > detected (hash mismatch) efi_attr_is_visible Hash = 567185240781730690
> > > [-Wbackend-plugin]
> > > arch/x86/crypto/aegis128-aesni-glue.c:265:30: warning: unused variable
> > > 'simd_alg' [-Wunused-variable]
> > > warning: lib/crypto/sha256.c: Function control flow change detected
> > > (hash mismatch) sha256_update Hash = 744640996947387358
> > > [-Wbackend-plugin]
> > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > change detected (hash mismatch) memcmp Hash = 742261418966908927
> > > [-Wbackend-plugin]
> > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > change detected (hash mismatch) bcmp Hash = 742261418966908927
> > > [-Wbackend-plugin]
> > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > change detected (hash mismatch) strcmp Hash = 536873291001348520
> > > [-Wbackend-plugin]
> > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > change detected (hash mismatch) strnlen Hash = 146835646621254984
> > > [-Wbackend-plugin]
> > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > change detected (hash mismatch) simple_strtoull Hash =
> > > 252792765950587360 [-Wbackend-plugin]
> > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > change detected (hash mismatch) strstr Hash = 391331303349076211
> > > [-Wbackend-plugin]
> > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > change detected (hash mismatch) strchr Hash = 1063705159280644635
> > > [-Wbackend-plugin]
> > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > change detected (hash mismatch) kstrtoull Hash = 758414239132790022
> > > [-Wbackend-plugin]
> > > drivers/infiniband/hw/hfi1/platform.o: warning: objtool: tune_serdes()
> > > falls through to next function apply_tx_lanes()
> > >
> > > Cannot say if this information is helpful.
> > >
> >
> > My LLVM/Clang v12 is from <apt.llvm.org>:
> >
> > clang-12 version 1:12~++20210115111113+45ef053bd709-1~exp1~20210115101809.3724
> >
> > My kernel-config is attached.
> >
>
> I dropped "LLVM_IAS=1" from my make line and did for my next build:
>
> $ scripts/diffconfig /boot/config-5.11.0-rc3-8-amd64-clang12-pgo .config
> BUILD_SALT "5.11.0-rc3-8-amd64-clang12-pgo" -> "5.11.0-rc3-10-amd64-clang12-pgo"
> DEBUG_INFO_DWARF2 n -> y
> DEBUG_INFO_DWARF5 y -> n
> PGO_CLANG y -> n
>
> Means dropped DWARF5 support.
>
Hi Sedat,

Using PGO just improves optimizations. So unless there's miscompile,
or some other nefarious thing, it shouldn't affect how the boot loader
runs.

As a sanity check, does the same Linux source and compiler version
generate a bootable kernel when PGO isn't used?

-bw

2021-01-17 23:50:17

by Sedat Dilek

[permalink] [raw]

Subject: Re: [PATCH v5] pgo: add clang's Profile Guided Optimization infrastructure

On Sun, Jan 17, 2021 at 9:35 PM Bill Wendling <[email protected]> wrote:
>
> On Sun, Jan 17, 2021 at 9:42 AM Sedat Dilek <[email protected]> wrote:
> >
> > On Sun, Jan 17, 2021 at 1:05 PM Sedat Dilek <[email protected]> wrote:
> > >
> > > On Sun, Jan 17, 2021 at 12:58 PM Sedat Dilek <[email protected]> wrote:
> > > >
> > > > On Sun, Jan 17, 2021 at 12:42 PM Sedat Dilek <[email protected]> wrote:
> > > > >
> > > > > On Sun, Jan 17, 2021 at 12:23 PM Sedat Dilek <[email protected]> wrote:
> > > > > >
> > > > > > On Sun, Jan 17, 2021 at 11:53 AM Sedat Dilek <[email protected]> wrote:
> > > > > > >
> > > > > > > On Sun, Jan 17, 2021 at 11:44 AM Sedat Dilek <[email protected]> wrote:
> > > > > > > >
> > > > > > > > On Sat, Jan 16, 2021 at 9:23 PM Bill Wendling <[email protected]> wrote:
> > > > > > > > >
> > > > > > > > > On Sat, Jan 16, 2021 at 9:39 AM Sedat Dilek <[email protected]> wrote:
> > > > > > > > > > On Sat, Jan 16, 2021 at 10:44 AM 'Bill Wendling' via Clang Built Linux
> > > > > > > > > > <[email protected]> wrote:
> > > > > > > > > > >
> > > > > > > > > > > From: Sami Tolvanen <[email protected]>
> > > > > > > > > > >
> > > > > > > > > > > Enable the use of clang's Profile-Guided Optimization[1]. To generate a
> > > > > > > > > > > profile, the kernel is instrumented with PGO counters, a representative
> > > > > > > > > > > workload is run, and the raw profile data is collected from
> > > > > > > > > > > /sys/kernel/debug/pgo/profraw.
> > > > > > > > > > >
> > > > > > > > > > > The raw profile data must be processed by clang's "llvm-profdata" tool
> > > > > > > > > > > before it can be used during recompilation:
> > > > > > > > > > >
> > > > > > > > > > > $ cp /sys/kernel/debug/pgo/profraw vmlinux.profraw
> > > > > > > > > > > $ llvm-profdata merge --output=vmlinux.profdata vmlinux.profraw
> > > > > > > > > > >
> > > > > > > > > > > Multiple raw profiles may be merged during this step.
> > > > > > > > > > >
> > > > > > > > > > > The data can now be used by the compiler:
> > > > > > > > > > >
> > > > > > > > > > > $ make LLVM=1 KCFLAGS=-fprofile-use=vmlinux.profdata ...
> > > > > > > > > > >
> > > > > > > > > > > This initial submission is restricted to x86, as that's the platform we
> > > > > > > > > > > know works. This restriction can be lifted once other platforms have
> > > > > > > > > > > been verified to work with PGO.
> > > > > > > > > > >
> > > > > > > > > > > Note that this method of profiling the kernel is clang-native, unlike
> > > > > > > > > > > the clang support in kernel/gcov.
> > > > > > > > > > >
> > > > > > > > > > > [1] https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization
> > > > > > > > > > >
> > > > > > > > > > > Signed-off-by: Sami Tolvanen <[email protected]>
> > > > > > > > > > > Co-developed-by: Bill Wendling <[email protected]>
> > > > > > > > > > > Signed-off-by: Bill Wendling <[email protected]>
> > > > > > > > > > > ---
> > > > > > > > > > > v2: - Added "__llvm_profile_instrument_memop" based on Nathan Chancellor's
> > > > > > > > > > > testing.
> > > > > > > > > > > - Corrected documentation, re PGO flags when using LTO, based on Fangrui
> > > > > > > > > > > Song's comments.
> > > > > > > > > > > v3: - Added change log section based on Sedat Dilek's comments.
> > > > > > > > > > > v4: - Remove non-x86 Makfile changes and se "hweight64" instead of using our
> > > > > > > > > > > own popcount implementation, based on Nick Desaulniers's comment.
> > > > > > > > > > > v5: - Correct padding calculation, discovered by Nathan Chancellor.
> > > > > > > > > > > ---
> > > > > > > > > > > Documentation/dev-tools/index.rst | 1 +
> > > > > > > > > > > Documentation/dev-tools/pgo.rst | 127 +++++++++
> > > > > > > > > > > MAINTAINERS | 9 +
> > > > > > > > > > > Makefile | 3 +
> > > > > > > > > > > arch/Kconfig | 1 +
> > > > > > > > > > > arch/x86/Kconfig | 1 +
> > > > > > > > > > > arch/x86/boot/Makefile | 1 +
> > > > > > > > > > > arch/x86/boot/compressed/Makefile | 1 +
> > > > > > > > > > > arch/x86/crypto/Makefile | 2 +
> > > > > > > > > > > arch/x86/entry/vdso/Makefile | 1 +
> > > > > > > > > > > arch/x86/kernel/vmlinux.lds.S | 2 +
> > > > > > > > > > > arch/x86/platform/efi/Makefile | 1 +
> > > > > > > > > > > arch/x86/purgatory/Makefile | 1 +
> > > > > > > > > > > arch/x86/realmode/rm/Makefile | 1 +
> > > > > > > > > > > arch/x86/um/vdso/Makefile | 1 +
> > > > > > > > > > > drivers/firmware/efi/libstub/Makefile | 1 +
> > > > > > > > > > > include/asm-generic/vmlinux.lds.h | 44 +++
> > > > > > > > > > > kernel/Makefile | 1 +
> > > > > > > > > > > kernel/pgo/Kconfig | 35 +++
> > > > > > > > > > > kernel/pgo/Makefile | 5 +
> > > > > > > > > > > kernel/pgo/fs.c | 382 ++++++++++++++++++++++++++
> > > > > > > > > > > kernel/pgo/instrument.c | 185 +++++++++++++
> > > > > > > > > > > kernel/pgo/pgo.h | 206 ++++++++++++++
> > > > > > > > > > > scripts/Makefile.lib | 10 +
> > > > > > > > > > > 24 files changed, 1022 insertions(+)
> > > > > > > > > > > create mode 100644 Documentation/dev-tools/pgo.rst
> > > > > > > > > > > create mode 100644 kernel/pgo/Kconfig
> > > > > > > > > > > create mode 100644 kernel/pgo/Makefile
> > > > > > > > > > > create mode 100644 kernel/pgo/fs.c
> > > > > > > > > > > create mode 100644 kernel/pgo/instrument.c
> > > > > > > > > > > create mode 100644 kernel/pgo/pgo.h
> > > > > > > > > > >
> > > > > > > > > > > diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/index.rst
> > > > > > > > > > > index f7809c7b1ba9e..8d6418e858062 100644
> > > > > > > > > > > --- a/Documentation/dev-tools/index.rst
> > > > > > > > > > > +++ b/Documentation/dev-tools/index.rst
> > > > > > > > > > > @@ -26,6 +26,7 @@ whole; patches welcome!
> > > > > > > > > > > kgdb
> > > > > > > > > > > kselftest
> > > > > > > > > > > kunit/index
> > > > > > > > > > > + pgo
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > .. only:: subproject and html
> > > > > > > > > > > diff --git a/Documentation/dev-tools/pgo.rst b/Documentation/dev-tools/pgo.rst
> > > > > > > > > > > new file mode 100644
> > > > > > > > > > > index 0000000000000..b7f11d8405b73
> > > > > > > > > > > --- /dev/null
> > > > > > > > > > > +++ b/Documentation/dev-tools/pgo.rst
> > > > > > > > > > > @@ -0,0 +1,127 @@
> > > > > > > > > > > +.. SPDX-License-Identifier: GPL-2.0
> > > > > > > > > > > +
> > > > > > > > > > > +===============================
> > > > > > > > > > > +Using PGO with the Linux kernel
> > > > > > > > > > > +===============================
> > > > > > > > > > > +
> > > > > > > > > > > +Clang's profiling kernel support (PGO_) enables profiling of the Linux kernel
> > > > > > > > > > > +when building with Clang. The profiling data is exported via the ``pgo``
> > > > > > > > > > > +debugfs directory.
> > > > > > > > > > > +
> > > > > > > > > > > +.. _PGO: https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization
> > > > > > > > > > > +
> > > > > > > > > > > +
> > > > > > > > > > > +Preparation
> > > > > > > > > > > +===========
> > > > > > > > > > > +
> > > > > > > > > > > +Configure the kernel with:
> > > > > > > > > > > +
> > > > > > > > > > > +.. code-block:: make
> > > > > > > > > > > +
> > > > > > > > > > > + CONFIG_DEBUG_FS=y
> > > > > > > > > > > + CONFIG_PGO_CLANG=y
> > > > > > > > > > > +
> > > > > > > > > > > +Note that kernels compiled with profiling flags will be significantly larger
> > > > > > > > > > > +and run slower.
> > > > > > > > > > > +
> > > > > > > > > > > +Profiling data will only become accessible once debugfs has been mounted:
> > > > > > > > > > > +
> > > > > > > > > > > +.. code-block:: sh
> > > > > > > > > > > +
> > > > > > > > > > > + mount -t debugfs none /sys/kernel/debug
> > > > > > > > > > > +
> > > > > > > > > > > +
> > > > > > > > > > > +Customization
> > > > > > > > > > > +=============
> > > > > > > > > > > +
> > > > > > > > > > > +You can enable or disable profiling for individual file and directories by
> > > > > > > > > > > +adding a line similar to the following to the respective kernel Makefile:
> > > > > > > > > > > +
> > > > > > > > > > > +- For a single file (e.g. main.o)
> > > > > > > > > > > +
> > > > > > > > > > > + .. code-block:: make
> > > > > > > > > > > +
> > > > > > > > > > > + PGO_PROFILE_main.o := y
> > > > > > > > > > > +
> > > > > > > > > > > +- For all files in one directory
> > > > > > > > > > > +
> > > > > > > > > > > + .. code-block:: make
> > > > > > > > > > > +
> > > > > > > > > > > + PGO_PROFILE := y
> > > > > > > > > > > +
> > > > > > > > > > > +To exclude files from being profiled use
> > > > > > > > > > > +
> > > > > > > > > > > + .. code-block:: make
> > > > > > > > > > > +
> > > > > > > > > > > + PGO_PROFILE_main.o := n
> > > > > > > > > > > +
> > > > > > > > > > > +and
> > > > > > > > > > > +
> > > > > > > > > > > + .. code-block:: make
> > > > > > > > > > > +
> > > > > > > > > > > + PGO_PROFILE := n
> > > > > > > > > > > +
> > > > > > > > > > > +Only files which are linked to the main kernel image or are compiled as kernel
> > > > > > > > > > > +modules are supported by this mechanism.
> > > > > > > > > > > +
> > > > > > > > > > > +
> > > > > > > > > > > +Files
> > > > > > > > > > > +=====
> > > > > > > > > > > +
> > > > > > > > > > > +The PGO kernel support creates the following files in debugfs:
> > > > > > > > > > > +
> > > > > > > > > > > +``/sys/kernel/debug/pgo``
> > > > > > > > > > > + Parent directory for all PGO-related files.
> > > > > > > > > > > +
> > > > > > > > > > > +``/sys/kernel/debug/pgo/reset``
> > > > > > > > > > > + Global reset file: resets all coverage data to zero when written to.
> > > > > > > > > > > +
> > > > > > > > > > > +``/sys/kernel/debug/profraw``
> > > > > > > > > > > + The raw PGO data that must be processed with ``llvm_profdata``.
> > > > > > > > > > > +
> > > > > > > > > > > +
> > > > > > > > > > > +Workflow
> > > > > > > > > > > +========
> > > > > > > > > > > +
> > > > > > > > > > > +The PGO kernel can be run on the host or test machines. The data though should
> > > > > > > > > > > +be analyzed with Clang's tools from the same Clang version as the kernel was
> > > > > > > > > > > +compiled. Clang's tolerant of version skew, but it's easier to use the same
> > > > > > > > > > > +Clang version.
> > > > > > > > > > > +
> > > > > > > > > > > +The profiling data is useful for optimizing the kernel, analyzing coverage,
> > > > > > > > > > > +etc. Clang offers tools to perform these tasks.
> > > > > > > > > > > +
> > > > > > > > > > > +Here is an example workflow for profiling an instrumented kernel with PGO and
> > > > > > > > > > > +using the result to optimize the kernel:
> > > > > > > > > > > +
> > > > > > > > > > > +1) Install the kernel on the TEST machine.
> > > > > > > > > > > +
> > > > > > > > > > > +2) Reset the data counters right before running the load tests
> > > > > > > > > > > +
> > > > > > > > > > > + .. code-block:: sh
> > > > > > > > > > > +
> > > > > > > > > > > + $ echo 1 > /sys/kernel/debug/pgo/reset
> > > > > > > > > > > +
> > > > > > > > > >
> > > > > > > > > > I do not get this...
> > > > > > > > > >
> > > > > > > > > > # mount | grep debugfs
> > > > > > > > > > debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
> > > > > > > > > >
> > > > > > > > > > After the load-test...?
> > > > > > > > > >
> > > > > > > > > > echo 0 > /sys/kernel/debug/pgo/reset
> > > > > > > > > >
> > > > > > > > > Writing anything to /sys/kernel/debug/pgo/reset will cause it to reset
> > > > > > > > > the profiling counters. I picked 1 (one) semi-randomly, but it could
> > > > > > > > > be any number, letter, your favorite short story, etc. You don't want
> > > > > > > > > to reset it before collecting the profiling data from your load tests
> > > > > > > > > though.
> > > > > > > > >
> > > > > > > > > > > +3) Run the load tests.
> > > > > > > > > > > +
> > > > > > > > > > > +4) Collect the raw profile data
> > > > > > > > > > > +
> > > > > > > > > > > + .. code-block:: sh
> > > > > > > > > > > +
> > > > > > > > > > > + $ cp -a /sys/kernel/debug/pgo/profraw /tmp/vmlinux.profraw
> > > > > > > > > > > +
> > > > > > > > > >
> > > > > > > > > > This is only 4,9M small and seen from the date 5mins before I run the
> > > > > > > > > > echo-1 line.
> > > > > > > > > >
> > > > > > > > > > # ll /sys/kernel/debug/pgo
> > > > > > > > > > insgesamt 0
> > > > > > > > > > drwxr-xr-x 2 root root 0 16. Jan 17:29 .
> > > > > > > > > > drwx------ 41 root root 0 16. Jan 17:29 ..
> > > > > > > > > > -rw------- 1 root root 0 16. Jan 17:29 profraw
> > > > > > > > > > --w------- 1 root root 0 16. Jan 18:19 reset
> > > > > > > > > >
> > > > > > > > > > # cp -a /sys/kernel/debug/pgo/profraw /tmp/vmlinux.profraw
> > > > > > > > > >
> > > > > > > > > > # ll /tmp/vmlinux.profraw
> > > > > > > > > > -rw------- 1 root root 4,9M 16. Jan 17:29 /tmp/vmlinux.profraw
> > > > > > > > > >
> > > > > > > > > > For me there was no prof-data collected from my defconfig kernel-build.
> > > > > > > > > >
> > > > > > > > > The /sys/kernel/debug/pgo/profraw file is read-only. Nothing writes to
> > > > > > > > > it, not even the kernel. All it does is serialize the profiling
> > > > > > > > > counters from a memory location in the kernel into a format that
> > > > > > > > > LLVM's tools can understand.
> > > > > > > > >
> > > > > > > > > > > +5) (Optional) Download the raw profile data to the HOST machine.
> > > > > > > > > > > +
> > > > > > > > > > > +6) Process the raw profile data
> > > > > > > > > > > +
> > > > > > > > > > > + .. code-block:: sh
> > > > > > > > > > > +
> > > > > > > > > > > + $ llvm-profdata merge --output=vmlinux.profdata vmlinux.profraw
> > > > > > > > > > > +
> > > > > > > > > >
> > > > > > > > > > Is that executed in /path/to/linux/git?
> > > > > > > > > >
> > > > > > > > > The llvm-profdata tool is not in the linux source tree. You need to
> > > > > > > > > grab it from a clang distribution (or built from clang's git repo).
> > > > > > > > >
> > > > > > > > > > > + Note that multiple raw profile data files can be merged during this step.
> > > > > > > > > > > +
> > > > > > > > > > > +7) Rebuild the kernel using the profile data (PGO disabled)
> > > > > > > > > > > +
> > > > > > > > > > > + .. code-block:: sh
> > > > > > > > > > > +
> > > > > > > > > > > + $ make LLVM=1 KCFLAGS=-fprofile-use=vmlinux.profdata ...
> > > > > > > > > >
> > > > > > > > > > How big is vmlinux.profdata (make defconfig)?
> > > > > > > > > >
> > > > > > > > > I don't have numbers for this, but from what you listed here, it's ~5M
> > > > > > > > > in size. The size is proportional to the number of counters
> > > > > > > > > instrumented in the kernel.
> > > > > > > > >
> > > > > > > > > > Do I need to do a full defconfig build or can I stop the build after
> > > > > > > > > > let me say 10mins?
> > > > > > > > > >
> > > > > > > > > You should do a full rebuild. Make sure that PGO is disabled during the rebuild.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Thanks Bill for all the information.
> > > > > > > >
> > > > > > > > And sorry if I am so pedantic.
> > > > > > > >
> > > > > > > > I have installed my Debian system with Legacy-BIOS enabled.
> > > > > > > >
> > > > > > > > When I rebuild with KCFLAGS=-fprofile-use=vmlinux.profdata (LLVM=1 I
> > > > > > > > have as a default) my system hangs on reboot.
> > > > > > > >
> > > > > > > > [ diffconfig ]
> > > > > > > > $ scripts/diffconfig /boot/config-5.11.0-rc3-8-amd64-clang12-pgo
> > > > > > > > /boot/config-5.11.0-rc3-9-amd64-clang12-pgo
> > > > > > > > BUILD_SALT "5.11.0-rc3-8-amd64-clang12-pgo" -> "5.11.0-rc3-9-amd64-clang12-pgo"
> > > > > > > > PGO_CLANG y -> n
> > > > > > > >
> > > > > > > > [ my make line ]
> > > > > > > > $ cat ../start-build_5.11.0-rc3-9-amd64-clang12-pgo.txt
> > > > > > > > dileks 63120 63095 0 06:47 pts/2 00:00:00 /usr/bin/perf_5.10
> > > > > > > > stat make V=1 -j4 HOSTCC=clang HOSTCXX=clang++ HOSTLD=ld.lld CC=clang
> > > > > > > > LD=ld.lld LLVM=1 LLVM_IAS=1 PAHOLE=/opt/pahole/bin/pahole
> > > > > > > > LOCALVERSION=-9-amd64-clang12-pgo KBUILD_VERBOSE=1
> > > > > > > > KBUILD_BUILD_HOST=iniza [email protected]
> > > > > > > > KBUILD_BUILD_TIMESTAMP=2021-01-17 bindeb-pkg
> > > > > > > > KDEB_PKGVERSION=5.11.0~rc3-9~bullseye+dileks1
> > > > > > > > KCFLAGS=-fprofile-use=vmlinux.profdata
> > > > > > > >
> > > > > > > > ( Yes, 06:47 a.m. in the morning :-). )
> > > > > > > >
> > > > > > > > When I boot with the rebuild Linux-kernel I see:
> > > > > > > >
> > > > > > > > Wrong EFI loader signature
> > > > > > > > ...
> > > > > > > > Decompressing
> > > > > > > > Parsing EFI
> > > > > > > > Performing Relocations done.
> > > > > > > > Booting the Kernel.
> > > > > > > >
> > > > > > > > *** SYSTEM HANGS ***
> > > > > > > > ( I waited for approx 1 min )
> > > > > > > >
> > > > > > > > I tried to turn UEFI support ON and OFF.
> > > > > > > > No success.
> > > > > > > >
> > > > > > > > Does Clang-PGO support Legacy-BIOS or is something different wrong?
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > >
> > > > > > >
> > > > > > > My bootloader is GRUB.
> > > > > > >
> > > > > > > In UEFI-BIOS settings there is no secure-boot disable option.
> > > > > > > Just simple "Use UEFI BIOS" enabled|disabled.
> > > > > > >
> > > > > > > Installed Debian packages:
> > > > > > >
> > > > > > > ii grub-common 2.04-12
> > > > > > > ii grub-pc 2.04-12
> > > > > > > ii grub-pc-bin 2.04-12
> > > > > > > ii grub2-common 2.04-12
> > > > > > >
> > > > > > > I found in the below link to do in grub-shell:
> > > > > > >
> > > > > > > set check_signatures=no
> > > > > > >
> > > > > > > But this is when grub-efi is installed.
> > > > > > >
> > > > > > > - Sedat -
> > > > > > >
> > > > > > > Link: https://unix.stackexchange.com/questions/126286/grub-efi-disable-signature-check
> > > > > >
> > > > > > Forget about that "Wrong EFI bootloader" - I see this with all other
> > > > > > kernels (all boot fine).
> > > > > >
> > > > > > I tried in QEMU with and without KASLR:
> > > > > >
> > > > > > [ run_qemu.sh ]
> > > > > > KPATH=$(pwd)
> > > > > >
> > > > > > APPEND="root=/dev/ram0 console=ttyS0 hung_task_panic=1 earlyprintk=ttyS0,115200"
> > > > > > APPEND="$APPEND nokaslr"
> > > > > >
> > > > > > qemu-system-x86_64 -enable-kvm -M pc -kernel $KPATH/bzImage -initrd
> > > > > > $KPATH/initrd.img -m 512 -net none -serial stdio -append "${APPEND}"
> > > > > > [ /run_qemu.sh ]
> > > > > >
> > > > > > $ ./run_qemu.sh
> > > > > > Probing EDD (edd=off to disable)... ok
> > > > > > Wrong EFI loader signature.
> > > > > > early console in extract_kernel
> > > > > > input_data: 0x000000000289940d
> > > > > > input_len: 0x000000000069804a
> > > > > > output: 0x0000000001000000
> > > > > > output_len: 0x0000000001ef2010
> > > > > > kernel_total_size: 0x0000000001c2c000
> > > > > > needed_size: 0x0000000002000000
> > > > > > trampoline_32bit: 0x000000000009d000
> > > > > >
> > > > > >
> > > > > > KASLR disabled: 'nokaslr' on cmdline.
> > > > > >
> > > > > >
> > > > > > Decompressing Linux... Parsing ELF... No relocation needed... done.
> > > > > > Booting the kernel.
> > > > > >
> > > > > > QEMU run stops, too.
> > > > > >
> > > > >
> > > > > I re-generated my initrd.img with GZIP as compressor (my default is ZSTD).
> > > > >
> > > > > --- /etc/initramfs-tools/initramfs.conf 2021-01-17 12:35:30.823818501 +0100
> > > > > +++ /etc/initramfs-tools/initramfs.conf.zstd 2020-09-21
> > > > > 23:55:43.121735427 +0200
> > > > > @@ -41,7 +41,7 @@ KEYMAP=n
> > > > > # COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz | zstd ]
> > > > > #
> > > > >
> > > > > -COMPRESS=gzip
> > > > > +COMPRESS=zstd
> > > > >
> > > > > #
> > > > > # DEVICE: ...
> > > > >
> > > > > root# KVER="5.11.0-rc3-9-amd64-clang12-pgo" ; update-initramfs -c -k $KVER
> > > > >
> > > > > QEMU boot stops at the same stage.
> > > > >
> > > > > Now, my head is empty...
> > > > >
> > > > > Any comments?
> > > > >
> > > >
> > > > ( Just as a side note I have Nick's DWARF-v5 support enabled. )
> > > >
> > > > There is one EFI related warning in my build-log:
> > > >
> > > > $ grep warning: build-log_5.11.0-rc3-9-amd64-clang12-pgo.txt
> > > > dpkg-architecture: warning: specified GNU system type x86_64-linux-gnu
> > > > does not match CC system type x86_64-pc-linux-gnu, try setting a
> > > > correct CC environment variable
> > > > warning: arch/x86/platform/efi/quirks.c: Function control flow change
> > > > detected (hash mismatch) efi_arch_mem_reserve Hash =
> > > > 391331300655996873 [-Wbackend-plugin]
> > > > warning: arch/x86/platform/efi/efi.c: Function control flow change
> > > > detected (hash mismatch) efi_attr_is_visible Hash = 567185240781730690
> > > > [-Wbackend-plugin]
> > > > arch/x86/crypto/aegis128-aesni-glue.c:265:30: warning: unused variable
> > > > 'simd_alg' [-Wunused-variable]
> > > > warning: lib/crypto/sha256.c: Function control flow change detected
> > > > (hash mismatch) sha256_update Hash = 744640996947387358
> > > > [-Wbackend-plugin]
> > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > change detected (hash mismatch) memcmp Hash = 742261418966908927
> > > > [-Wbackend-plugin]
> > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > change detected (hash mismatch) bcmp Hash = 742261418966908927
> > > > [-Wbackend-plugin]
> > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > change detected (hash mismatch) strcmp Hash = 536873291001348520
> > > > [-Wbackend-plugin]
> > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > change detected (hash mismatch) strnlen Hash = 146835646621254984
> > > > [-Wbackend-plugin]
> > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > change detected (hash mismatch) simple_strtoull Hash =
> > > > 252792765950587360 [-Wbackend-plugin]
> > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > change detected (hash mismatch) strstr Hash = 391331303349076211
> > > > [-Wbackend-plugin]
> > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > change detected (hash mismatch) strchr Hash = 1063705159280644635
> > > > [-Wbackend-plugin]
> > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > change detected (hash mismatch) kstrtoull Hash = 758414239132790022
> > > > [-Wbackend-plugin]
> > > > drivers/infiniband/hw/hfi1/platform.o: warning: objtool: tune_serdes()
> > > > falls through to next function apply_tx_lanes()
> > > >
> > > > Cannot say if this information is helpful.
> > > >
> > >
> > > My LLVM/Clang v12 is from <apt.llvm.org>:
> > >
> > > clang-12 version 1:12~++20210115111113+45ef053bd709-1~exp1~20210115101809.3724
> > >
> > > My kernel-config is attached.
> > >
> >
> > I dropped "LLVM_IAS=1" from my make line and did for my next build:
> >
> > $ scripts/diffconfig /boot/config-5.11.0-rc3-8-amd64-clang12-pgo .config
> > BUILD_SALT "5.11.0-rc3-8-amd64-clang12-pgo" -> "5.11.0-rc3-10-amd64-clang12-pgo"
> > DEBUG_INFO_DWARF2 n -> y
> > DEBUG_INFO_DWARF5 y -> n
> > PGO_CLANG y -> n
> >
> > Means dropped DWARF5 support.
> >
> Hi Sedat,
>
> Using PGO just improves optimizations. So unless there's miscompile,
> or some other nefarious thing, it shouldn't affect how the boot loader
> runs.
>
> As a sanity check, does the same Linux source and compiler version
> generate a bootable kernel when PGO isn't used?
>

Yes, I can boot with the same code base without PGO.

With the attached kernel-config.

I remember there is a fix in CBL issue tracker for...

( https://github.com/ClangBuiltLinux/linux/issues/1250 )

Loading, please wait...
Starting version 247.2-4
[ 2.157223] floppy: module verification failed: signature and/or
required key missing - tainting kernel
[ 2.179326] i2c_piix4: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
[ 2.183558] scsi_mod: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
[ 2.187991] floppy: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
[ 2.195047] psmouse: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
[ 2.210404] scsi_mod: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
[ 2.231055] scsi_mod: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)

Full QEMU log...

$ ./run_qemu.sh
Probing EDD (edd=off to disable)... ok
Wrong EFI loader signature.
early console in extract_kernel
input_data: 0x000000000289c40d
input_len: 0x0000000000693f62
output: 0x0000000001000000
output_len: 0x0000000001ef0224
kernel_total_size: 0x0000000001c2c000
needed_size: 0x0000000002000000
trampoline_32bit: 0x000000000009d000
Physical KASLR using RDTSC...
Virtual KASLR using RDTSC...

Decompressing Linux... Parsing ELF... Performing relocations... done.
Booting the kernel.
[ 0.000000] Linux version 5.11.0-rc3-10-amd64-clang12-pgo
([email protected]@iniza) (Debian clang version
12.0.0-++20210115111113+45ef053bd709-1~exp1~2021011510
1809.3724, LLD 12.0.0) #10~bullseye+dileks1 SMP 2021-01-17
[ 0.000000] Command line: root=/dev/ram0 console=ttyS0
hung_task_panic=1 earlyprintk=ttyS0,115200
[ 0.000000] x86/fpu: x87 FPU will use FXSAVE
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000001ffdffff] usable
[ 0.000000] BIOS-e820: [mem 0x000000001ffe0000-0x000000001fffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[ 0.000000] printk: bootconsole [earlyser0] enabled
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.8 present.
[ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.14.0-2 04/01/2014
[ 0.000000] Hypervisor detected: KVM
[ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[ 0.000000] kvm-clock: cpu 0, msr c877001, primary cpu clock
[ 0.000000] kvm-clock: using sched offset of 510123624 cycles
[ 0.003240] clocksource: kvm-clock: mask: 0xffffffffffffffff
max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[ 0.009652] tsc: Detected 1596.372 MHz processor
[ 0.013107] last_pfn = 0x1ffe0 max_arch_pfn = 0x400000000
[ 0.015537] x86/PAT: PAT not supported by the CPU.
[ 0.017605] x86/PAT: Configuration [0-7]: WB WT UC- UC WB WT UC- UC
Memory KASLR using RDTSC...
[ 0.038444] found SMP MP-table at [mem 0x000f5ce0-0x000f5cef]
[ 0.042330] RAMDISK: [mem 0x1dfdb000-0x1ffdffff]
[ 0.044738] ACPI: Early table checksum verification disabled
[ 0.047289] ACPI: RSDP 0x00000000000F5B20 000014 (v00 BOCHS )
[ 0.049887] ACPI: RSDT 0x000000001FFE1550 000034 (v01 BOCHS
BXPCRSDT 00000001 BXPC 00000001)
[ 0.054578] ACPI: FACP 0x000000001FFE1404 000074 (v01 BOCHS
BXPCFACP 00000001 BXPC 00000001)
[ 0.058412] ACPI: DSDT 0x000000001FFE0040 0013C4 (v01 BOCHS
BXPCDSDT 00000001 BXPC 00000001)
[ 0.062056] ACPI: FACS 0x000000001FFE0000 000040
[ 0.064325] ACPI: APIC 0x000000001FFE1478 000078 (v01 BOCHS
BXPCAPIC 00000001 BXPC 00000001)
[ 0.068546] ACPI: HPET 0x000000001FFE14F0 000038 (v01 BOCHS
BXPCHPET 00000001 BXPC 00000001)
[ 0.073026] ACPI: WAET 0x000000001FFE1528 000028 (v01 BOCHS
BXPCWAET 00000001 BXPC 00000001)
[ 0.078063] No NUMA configuration found
[ 0.080007] Faking a node at [mem 0x0000000000000000-0x000000001ffdffff]
[ 0.083430] NODE_DATA(0) allocated [mem 0x1dfb1000-0x1dfdafff]
[ 0.086934] Zone ranges:
[ 0.087919] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[ 0.089927] DMA32 [mem 0x0000000001000000-0x000000001ffdffff]
[ 0.092270] Normal empty
[ 0.093824] Device empty
[ 0.095069] Movable zone start for each node
[ 0.096880] Early memory node ranges
[ 0.098410] node 0: [mem 0x0000000000001000-0x000000000009efff]
[ 0.101939] node 0: [mem 0x0000000000100000-0x000000001ffdffff]
[ 0.106130] Zeroed struct page in unavailable ranges: 130 pages
[ 0.106139] Initmem setup node 0 [mem 0x0000000000001000-0x000000001ffdffff]
[ 0.115094] ACPI: PM-Timer IO Port: 0x608
[ 0.117173] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[ 0.121073] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
[ 0.123537] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.126254] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[ 0.129062] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 0.131888] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[ 0.135065] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[ 0.137286] Using ACPI (MADT) for SMP configuration information
[ 0.139743] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[ 0.141956] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[ 0.143678] PM: hibernation: Registered nosave memory: [mem
0x00000000-0x00000fff]
[ 0.146249] PM: hibernation: Registered nosave memory: [mem
0x0009f000-0x0009ffff]
[ 0.148784] PM: hibernation: Registered nosave memory: [mem
0x000a0000-0x000effff]
[ 0.152756] PM: hibernation: Registered nosave memory: [mem
0x000f0000-0x000fffff]
[ 0.155969] [mem 0x20000000-0xfeffbfff] available for PCI devices
[ 0.158542] Booting paravirtualized kernel on KVM
[ 0.160520] clocksource: refined-jiffies: mask: 0xffffffff
max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[ 0.171049] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:1
nr_cpu_ids:1 nr_node_ids:1
[ 0.175162] percpu: Embedded 54 pages/cpu s183512 r8192 d29480 u2097152
[ 0.178044] kvm-guest: stealtime: cpu 0, msr 1d418480
[ 0.180197] kvm-guest: PV spinlocks disabled, no host support
[ 0.182655] Built 1 zonelists, mobility grouping on. Total pages: 128872
[ 0.188717] Policy zone: DMA32
[ 0.190055] Kernel command line: root=/dev/ram0 console=ttyS0
hung_task_panic=1 earlyprintk=ttyS0,115200
[ 0.194307] Dentry cache hash table entries: 65536 (order: 7,
524288 bytes, linear)
[ 0.197691] Inode-cache hash table entries: 32768 (order: 6, 262144
bytes, linear)
[ 0.201953] mem auto-init: stack:off, heap alloc:on, heap free:off
[ 0.206787] Memory: 232680K/523768K available (12295K kernel code,
2462K rwdata, 4008K rodata, 2444K init, 1888K bss, 71012K reserved, 0K
cma-reserved)
[ 0.212719] random: get_random_u64 called from
kmem_cache_open+0x27/0x500 with crng_init=0
[ 0.212892] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.220858] Kernel/User page tables isolation: enabled
[ 0.223136] ftrace: allocating 36189 entries in 142 pages
[ 0.249721] ftrace: allocated 142 pages with 4 groups
[ 0.252993] rcu: Hierarchical RCU implementation.
[ 0.255411] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=1.
[ 0.258890] Rude variant of Tasks RCU enabled.
[ 0.260761] Tracing variant of Tasks RCU enabled.
[ 0.262625] rcu: RCU calculated value of scheduler-enlistment delay
is 25 jiffies.
[ 0.265212] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[ 0.271882] NR_IRQS: 524544, nr_irqs: 256, preallocated irqs: 16
[ 0.295378] Console: colour VGA+ 80x25
[ 0.297439] printk: console [ttyS0] enabled
[ 0.297439] printk: console [ttyS0] enabled
[ 0.302560] printk: bootconsole [earlyser0] disabled
[ 0.302560] printk: bootconsole [earlyser0] disabled
[ 0.307728] ACPI: Core revision 20201113
[ 0.310172] clocksource: hpet: mask: 0xffffffff max_cycles:
0xffffffff, max_idle_ns: 19112604467 ns
[ 0.315115] APIC: Switch to symmetric I/O mode setup
[ 0.318899] x2apic enabled
[ 0.321088] Switched APIC routing to physical x2apic.
[ 0.326381] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.329537] clocksource: tsc-early: mask: 0xffffffffffffffff
max_cycles: 0x1702c1d9d3d, max_idle_ns: 440795278546 ns
[ 0.335417] Calibrating delay loop (skipped) preset value.. 3192.74
BogoMIPS (lpj=6385488)
[ 0.339418] pid_max: default: 32768 minimum: 301
[ 0.341620] LSM: Security Framework initializing
[ 0.343446] Yama: becoming mindful.
[ 0.345314] AppArmor: AppArmor initialized
[ 0.347421] TOMOYO Linux initialized
[ 0.349270] Mount-cache hash table entries: 1024 (order: 1, 8192
bytes, linear)
[ 0.351417] Mountpoint-cache hash table entries: 1024 (order: 1,
8192 bytes, linear)
Poking KASLR using RDTSC...
[ 0.361119] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
[ 0.363416] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
[ 0.367419] Spectre V1 : Mitigation: usercopy/swapgs barriers and
__user pointer sanitization
[ 0.370260] Spectre V2 : Mitigation: Full generic retpoline
[ 0.371412] Spectre V2 : Spectre v2 / SpectreRSB mitigation:
Filling RSB on context switch
[ 0.374257] Speculative Store Bypass: Vulnerable
[ 0.375416] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
[ 0.389948] Freeing SMP alternatives memory: 36K
[ 0.505617] APIC calibration not consistent with PM-Timer: 101ms
instead of 100ms
[ 0.507410] APIC delta adjusted to PM-Timer: 6252138 (6321934)
[ 0.507513] smpboot: CPU0: Intel QEMU Virtual CPU version 2.5+
(family: 0x6, model: 0x6, stepping: 0x3)
[ 0.512111] Performance Events: PMU not available due to
virtualization, using software events only.
[ 0.515510] rcu: Hierarchical SRCU implementation.
[ 0.517439] NMI watchdog: Perf NMI watchdog permanently disabled
[ 0.519477] smp: Bringing up secondary CPUs ...
[ 0.523416] smp: Brought up 1 node, 1 CPU
[ 0.525134] smpboot: Max logical packages: 1
[ 0.526969] smpboot: Total of 1 processors activated (3192.74 BogoMIPS)
[ 0.532118] node 0 deferred pages initialised in 4ms
[ 0.534052] devtmpfs: initialized
[ 0.535262] x86/mm: Memory block size: 128MB
[ 0.535711] clocksource: jiffies: mask: 0xffffffff max_cycles:
0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.539428] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.541875] pinctrl core: initialized pinctrl subsystem
[ 0.543936] NET: Registered protocol family 16
[ 0.547553] audit: initializing netlink subsys (disabled)
[ 0.551634] thermal_sys: Registered thermal governor 'fair_share'
[ 0.551637] thermal_sys: Registered thermal governor 'bang_bang'
[ 0.554723] thermal_sys: Registered thermal governor 'step_wise'
[ 0.555425] audit: type=2000 audit(1610926004.833:1):
state=initialized audit_enabled=0 res=1
[ 0.563420] thermal_sys: Registered thermal governor 'user_space'
[ 0.563434] cpuidle: using governor ladder
[ 0.569524] cpuidle: using governor menu
[ 0.571485] ACPI: bus type PCI registered
[ 0.573517] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[ 0.576324] PCI: Using configuration type 1 for base access
[ 0.580588] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.772658] ACPI: Added _OSI(Module Device)
[ 0.774521] ACPI: Added _OSI(Processor Device)
[ 0.775417] ACPI: Added _OSI(3.0 _SCP Extensions)
[ 0.778176] ACPI: Added _OSI(Processor Aggregator Device)
[ 0.779432] ACPI: Added _OSI(Linux-Dell-Video)
[ 0.783458] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[ 0.785480] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[ 0.788133] ACPI: 1 ACPI AML tables successfully acquired and loaded
[ 0.792295] ACPI: Interpreter enabled
[ 0.794716] ACPI: (supports S0 S3 S4 S5)
[ 0.795415] ACPI: Using IOAPIC for interrupt routing
[ 0.797540] PCI: Using host bridge windows from ACPI; if necessary,
use "pci=nocrs" and report a bug
[ 0.799590] ACPI: Enabled 2 GPEs in block 00 to 0F
[ 0.807844] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[ 0.811186] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM
Segments MSI HPX-Type3]
[ 0.811444] acpi PNP0A03:00: fail to add MMCONFIG information,
can't access extended PCI configuration space under this bridge.
[ 0.815895] acpiphp: Slot [3] registered
[ 0.819473] acpiphp: Slot [4] registered
[ 0.821210] acpiphp: Slot [5] registered
[ 0.823453] acpiphp: Slot [6] registered
[ 0.825153] acpiphp: Slot [7] registered
[ 0.827461] acpiphp: Slot [8] registered
[ 0.829166] acpiphp: Slot [9] registered
[ 0.831537] acpiphp: Slot [10] registered
[ 0.833276] acpiphp: Slot [11] registered
[ 0.835447] acpiphp: Slot [12] registered
[ 0.837183] acpiphp: Slot [13] registered
[ 0.839428] acpiphp: Slot [14] registered
[ 0.841167] acpiphp: Slot [15] registered
[ 0.843042] acpiphp: Slot [16] registered
[ 0.843455] acpiphp: Slot [17] registered
[ 0.845205] acpiphp: Slot [18] registered
[ 0.847452] acpiphp: Slot [19] registered
[ 0.849209] acpiphp: Slot [20] registered
[ 0.851448] acpiphp: Slot [21] registered
[ 0.853215] acpiphp: Slot [22] registered
[ 0.855447] acpiphp: Slot [23] registered
[ 0.857179] acpiphp: Slot [24] registered
[ 0.859478] acpiphp: Slot [25] registered
[ 0.861807] acpiphp: Slot [26] registered
[ 0.863150] acpiphp: Slot [27] registered
[ 0.863458] acpiphp: Slot [28] registered
[ 0.865444] acpiphp: Slot [29] registered
[ 0.867451] acpiphp: Slot [30] registered
[ 0.868826] acpiphp: Slot [31] registered
[ 0.870296] PCI host bridge to bus 0000:00
[ 0.871415] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]
[ 0.875414] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]
[ 0.879418] pci_bus 0000:00: root bus resource [mem
0x000a0000-0x000bffff window]
[ 0.883416] pci_bus 0000:00: root bus resource [mem
0x20000000-0xfebfffff window]
[ 0.887416] pci_bus 0000:00: root bus resource [mem
0x100000000-0x17fffffff window]
[ 0.891277] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 0.891510] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
[ 0.896375] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
[ 0.900672] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
[ 0.908157] pci 0000:00:01.1: reg 0x20: [io 0xc000-0xc00f]
[ 0.912723] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7]
[ 0.915417] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6]
[ 0.919413] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177]
[ 0.923413] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376]
[ 0.926608] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
[ 0.928431] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by
PIIX4 ACPI
[ 0.931433] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB
[ 0.935898] pci 0000:00:02.0: [1234:1111] type 00 class 0x030000
[ 0.941665] pci 0000:00:02.0: reg 0x10: [mem 0xfd000000-0xfdffffff pref]
[ 0.949458] pci 0000:00:02.0: reg 0x18: [mem 0xfebf0000-0xfebf0fff]
[ 0.958562] pci 0000:00:02.0: reg 0x30: [mem 0xfebe0000-0xfebeffff pref]
[ 0.961151] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
[ 0.963610] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[ 0.966032] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[ 0.967627] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
[ 0.971526] ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
[ 0.974667] iommu: Default domain type: Translated
[ 0.975568] pci 0000:00:02.0: vgaarb: setting as boot VGA device
[ 0.978113] pci 0000:00:02.0: vgaarb: VGA device added:
decodes=io+mem,owns=io+mem,locks=none
[ 0.979413] pci 0000:00:02.0: vgaarb: bridge control possible
[ 0.983413] vgaarb: loaded
[ 0.984827] EDAC MC: Ver: 3.0.0
[ 0.988222] NetLabel: Initializing
[ 0.991415] NetLabel: domain hash size = 128
[ 0.992873] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
[ 0.994825] NetLabel: unlabeled traffic allowed by default
[ 0.995430] PCI: Using ACPI for IRQ routing
[ 0.999490] hpet: 3 channels of 0 reserved for per-cpu timers
[ 1.001394] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[ 1.002975] hpet0: 3 comparators, 64-bit 100.000000 MHz counter
[ 1.009634] clocksource: Switched to clocksource kvm-clock
[ 1.019197] VFS: Disk quotas dquot_6.6.0
[ 1.021644] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 1.029347] AppArmor: AppArmor Filesystem Enabled
[ 1.031431] pnp: PnP ACPI init
[ 1.033294] pnp: PnP ACPI: found 6 devices
[ 1.041838] clocksource: acpi_pm: mask: 0xffffff max_cycles:
0xffffff, max_idle_ns: 2085701024 ns
[ 1.045506] NET: Registered protocol family 2
[ 1.047325] tcp_listen_portaddr_hash hash table entries: 256
(order: 0, 4096 bytes, linear)
[ 1.051250] TCP established hash table entries: 4096 (order: 3,
32768 bytes, linear)
[ 1.054797] TCP bind hash table entries: 4096 (order: 4, 65536 bytes, linear)
[ 1.057867] TCP: Hash tables configured (established 4096 bind 4096)
[ 1.060657] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[ 1.063438] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[ 1.066616] NET: Registered protocol family 1
[ 1.068525] NET: Registered protocol family 44
[ 1.070988] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]
[ 1.073088] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]
[ 1.075350] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[ 1.078851] pci_bus 0000:00: resource 7 [mem 0x20000000-0xfebfffff window]
[ 1.082396] pci_bus 0000:00: resource 8 [mem 0x100000000-0x17fffffff window]
[ 1.086505] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[ 1.089003] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[ 1.091193] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[ 1.093281] pci 0000:00:02.0: Video device with shadowed ROM at
[mem 0x000c0000-0x000dffff]
[ 1.096308] PCI: CLS 0 bytes, default 64
[ 1.098784] Trying to unpack rootfs image as initramfs...
[ 1.756924] Freeing initrd memory: 32788K
[ 1.759044] clocksource: tsc: mask: 0xffffffffffffffff max_cycles:
0x1702c1d9d3d, max_idle_ns: 440795278546 ns
[ 1.765351] Initialise system trusted keyrings
[ 1.767287] Key type blacklist registered
[ 1.769096] workingset: timestamp_bits=36 max_order=17 bucket_order=0
[ 1.773218] zbud: loaded
[ 1.774596] integrity: Platform Keyring initialized
[ 1.776709] Key type asymmetric registered
[ 1.779399] Asymmetric key parser 'x509' registered
[ 1.781504] Block layer SCSI generic (bsg) driver version 0.4
loaded (major 251)
[ 1.784737] io scheduler mq-deadline registered
[ 1.786842] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[ 1.790028] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 1.793393] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200)
is a 16550A
[ 1.798437] Linux agpgart interface v0.103
[ 1.799944] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <[email protected]>
[ 1.802535] AMD-Vi: AMD IOMMUv2 functionality not available on this system
[ 1.806358] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU]
at 0x60,0x64 irq 1,12
[ 1.810762] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 1.813927] serio: i8042 AUX port at 0x60,0x64 irq 12
[ 1.816306] mousedev: PS/2 mouse device common for all mice
[ 1.819254] input: AT Translated Set 2 keyboard as
/devices/platform/i8042/serio0/input/input0
[ 1.823023] rtc_cmos 00:05: RTC can wake from S4
[ 1.826320] rtc_cmos 00:05: registered as rtc0
[ 1.829030] rtc_cmos 00:05: setting system clock to
2021-01-17T23:26:45 UTC (1610926005)
[ 1.832489] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes
nvram, hpet irqs
[ 1.835661] intel_pstate: CPU model not supported
[ 1.837656] ledtrig-cpu: registered to indicate activity on CPUs
[ 1.840489] NET: Registered protocol family 10
[ 1.857135] Segment Routing with IPv6
[ 1.858772] mip6: Mobile IPv6
[ 1.860093] NET: Registered protocol family 17
[ 1.862844] mpls_gso: MPLS GSO support
[ 1.864379] IPI shorthand broadcast: enabled
[ 1.865844] sched_clock: Marking stable (1819436328,
44726425)->(1868284483, -4121730)
[ 1.869029] registered taskstats version 1
[ 1.870771] Loading compiled-in X.509 certificates
[ 1.873185] zswap: loaded using pool zstd/zbud
[ 1.875399] Key type ._fscrypt registered
[ 1.877158] Key type .fscrypt registered
[ 1.879447] Key type fscrypt-provisioning registered
[ 1.881189] AppArmor: AppArmor sha1 policy hashing enabled
[ 1.886920] Freeing unused kernel image (initmem) memory: 2444K
[ 1.891517] Write protecting the kernel read-only data: 18432k
[ 1.896049] Freeing unused kernel image (text/rodata gap) memory: 2040K
[ 1.899196] Freeing unused kernel image (rodata/data gap) memory: 88K
[ 1.968324] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[ 1.971797] x86/mm: Checking user space page tables
[ 2.037848] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[ 2.040258] Run /init as init process
Loading, please wait...
Starting version 247.2-4
[ 2.157223] floppy: module verification failed: signature and/or
required key missing - tainting kernel
[ 2.179326] i2c_piix4: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
[ 2.183558] scsi_mod: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
[ 2.187991] floppy: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
[ 2.195047] psmouse: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
[ 2.210404] scsi_mod: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
[ 2.231055] scsi_mod: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... [ 2.261574] libcrc32c:
Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
Scanning for Btrfs filesystems
done.
Begin: Waiting for root file system ... Begin: Running
/scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
qemu-system-x86_64: terminating on signal 2

- Sedat -

Attachments:

config-5.11.0-rc3-10-amd64-clang12-pgo (232.66 kB)

2021-01-18 00:41:58

Hi,

When I looked through the code I wondered why we do not add a
"CONFIG_PGO_CLANG_PROFDATA" which can be helpful when doing the PGO
rebuild with a vmlinux.profdata.

This introduces a "PGO_PROFDATA" to turn on/off to pass
"-fprofile-use=vmlinux.profdata" (see CFLAGS_PGO_CLANG_PROFDATA in
top-level Makefile).

If we turn off via "PGO_PROFILE := n" in several Makefiles - we should
do the same and add "PGO_PROFDATA := n" to the same Makefiles?

Please see the attached diff.

Regards,
- Sedat -

Attachments:

CONFIG_PGO_CLANG_PROFDATA.diff (4.66 kB)

2021-01-21 03:40:58

On Fri, Jan 22, 2021 at 2:44 AM Nick Desaulniers
<[email protected]> wrote:
>
> On Thu, Jan 21, 2021 at 2:35 AM Sedat Dilek <[email protected]> wrote:
> >
> > Thanks for v6.
> >
> > Small Changelog nits:
> > * Can you reverse-order the changelog - latest v6 first.
> > * v4: s/Makfile changes and se/Ma*k*efile changes and *u*se
> >
> > Can you add a hint to this "Clang-PGO" patch requiring Clang >= 12?
> >
> > Can you please add a comment for people using Clang >= 12 (ToT) and
> > have kernel-modules enabled, they will need the patch from CBL issue
> > #1250?
> > Otherwise they cannot boot and follow the next steps in the workflow.
> >
> > Can you put a comment about value "1" to reset the profiling counter?
> > That there is no "0" value stopping it.
> >
> > Can you add an example for the workload test?
> > Here I do a x86-64 defconfig build.
> > See attached script.
> >
> > Usually, I download this patch from LORE.
> >
> > link="https://lore.kernel.org/r/[email protected]"
> > b4 -d am $link
> >
> > This downloads v6.
> >
> > What if I want a previous version (compare)?
> > Again, I will love to see a "clang-pgo" branch and maybe tags for the
> > several versions in your personal GitHub.
> > Come on, Bill :-).
>
> That's quite a long list, Sedat! Do you think some of these can be
> follow ups, once the core lands? I'd much prefer to land the meat of
> this and follow up quickly, than tire out poor Bill! :P
>

Poor Bill - he lost his hairs :-)?

I hoped that the documentation gets a bit improved and clearer at some places.

- Sedat -

2021-01-22 02:38:54

On Mon, Jun 14, 2021 at 11:07 AM Nick Desaulniers
<[email protected]> wrote:
>
> On Mon, Jun 14, 2021 at 9:23 AM Kees Cook <[email protected]> wrote:
> >
> > On Mon, Jun 14, 2021 at 05:35:45PM +0200, Peter Zijlstra wrote:
> > > On Mon, Jun 14, 2021 at 08:26:01AM -0700, Kees Cook wrote:
> > > > So, AFAICT, the original blocking issue ("PGO does not respect noinstr")
> > > > is not actually an issue (noinstr contains notrace, which IS respected
> > > > by PGO[2]), I think this is fine to move forward.
> > >
> > > It is *NOT*: https://godbolt.org/z/9c7xdvGd9
> > >
> > > Look at how both compilers generate instrumentation in the no_instr()
> > > function.
> >
> > Well that's disappointing. I'll put this on hold until Clang can grow an
> > appropriate attribute (or similar work-around). Thanks for catching
> > that.
>
> Cross referencing since these two threads are related.
> https://lore.kernel.org/lkml/CAKwvOdmPTi93n2L0_yQkrzLdmpxzrOR7zggSzonyaw2PGshApw@mail.gmail.com/

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80223 looked appropriate
to me, so I commented on it.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80223#c6

Patches for:
PGO: https://reviews.llvm.org/D104253
GCOV: https://reviews.llvm.org/D104257
--
Thanks,
~Nick Desaulniers