GNU's addr2line can have problems parsing a vmlinux built with LLVM,
particularly when LTO was used. In order to decode the traces correctly
this patch adds the ability to switch to LLVM's utilities readelf and
addr2line. The same approach is followed by Will in [1].
Before:
$ scripts/decode_stacktrace.sh vmlinux < kernel.log
[17716.240635] Call trace:
[17716.240646] skb_cow_data (??:?)
[17716.240654] esp6_input (ld-temp.o:?)
[17716.240666] xfrm_input (ld-temp.o:?)
[17716.240674] xfrm6_rcv (??:?)
[...]
After:
$ LLVM=1 scripts/decode_stacktrace.sh vmlinux < kernel.log
[17716.240635] Call trace:
[17716.240646] skb_cow_data (include/linux/skbuff.h:2172 net/core/skbuff.c:4503)
[17716.240654] esp6_input (net/ipv6/esp6.c:977)
[17716.240666] xfrm_input (net/xfrm/xfrm_input.c:659)
[17716.240674] xfrm6_rcv (net/ipv6/xfrm6_input.c:172)
[...]
Note that one could set CROSS_COMPILE=llvm- instead to hack around this
issue. However, doing so can break the decodecode routine as it will
force the selection of other LLVM utilities down the line e.g. llvm-as.
[1] https://lore.kernel.org/all/[email protected]/
Cc: Will Deacon <[email protected]>
Cc: John Stultz <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Signed-off-by: Carlos Llamas <[email protected]>
---
scripts/decode_stacktrace.sh | 19 +++++++++++++++++--
1 file changed, 17 insertions(+), 2 deletions(-)
diff --git a/scripts/decode_stacktrace.sh b/scripts/decode_stacktrace.sh
index 564c5632e1a2..bfe5a4082d8e 100755
--- a/scripts/decode_stacktrace.sh
+++ b/scripts/decode_stacktrace.sh
@@ -16,6 +16,21 @@ elif type c++filt >/dev/null 2>&1 ; then
cppfilt_opts=-i
fi
+UTIL_SUFFIX=
+if [[ -z ${LLVM:-} ]]; then
+ UTIL_PREFIX=${CROSS_COMPILE:-}
+else
+ UTIL_PREFIX=llvm-
+ if [[ ${LLVM} == */ ]]; then
+ UTIL_PREFIX=${LLVM}${UTIL_PREFIX}
+ elif [[ ${LLVM} == -* ]]; then
+ UTIL_SUFFIX=${LLVM}
+ fi
+fi
+
+READELF=${UTIL_PREFIX}readelf${UTIL_SUFFIX}
+ADDR2LINE=${UTIL_PREFIX}addr2line${UTIL_SUFFIX}
+
if [[ $1 == "-r" ]] ; then
vmlinux=""
basepath="auto"
@@ -75,7 +90,7 @@ find_module() {
if [[ "$modpath" != "" ]] ; then
for fn in $(find "$modpath" -name "${module//_/[-_]}.ko*") ; do
- if readelf -WS "$fn" | grep -qwF .debug_line ; then
+ if ${READELF} -WS "$fn" | grep -qwF .debug_line ; then
echo $fn
return
fi
@@ -169,7 +184,7 @@ parse_symbol() {
if [[ $aarray_support == true && "${cache[$module,$address]+isset}" == "isset" ]]; then
local code=${cache[$module,$address]}
else
- local code=$(${CROSS_COMPILE}addr2line -i -e "$objfile" "$address" 2>/dev/null)
+ local code=$(${ADDR2LINE} -i -e "$objfile" "$address" 2>/dev/null)
if [[ $aarray_support == true ]]; then
cache[$module,$address]=$code
fi
--
2.42.0.582.g8ccd20d70d-goog
On Thu, Sep 28, 2023 at 8:48 PM Carlos Llamas <[email protected]> wrote:
>
> GNU's addr2line can have problems parsing a vmlinux built with LLVM,
> particularly when LTO was used. In order to decode the traces correctly
> this patch adds the ability to switch to LLVM's utilities readelf and
> addr2line. The same approach is followed by Will in [1].
>
> Before:
> $ scripts/decode_stacktrace.sh vmlinux < kernel.log
> [17716.240635] Call trace:
> [17716.240646] skb_cow_data (??:?)
> [17716.240654] esp6_input (ld-temp.o:?)
> [17716.240666] xfrm_input (ld-temp.o:?)
> [17716.240674] xfrm6_rcv (??:?)
> [...]
>
> After:
> $ LLVM=1 scripts/decode_stacktrace.sh vmlinux < kernel.log
> [17716.240635] Call trace:
> [17716.240646] skb_cow_data (include/linux/skbuff.h:2172 net/core/skbuff.c:4503)
> [17716.240654] esp6_input (net/ipv6/esp6.c:977)
> [17716.240666] xfrm_input (net/xfrm/xfrm_input.c:659)
> [17716.240674] xfrm6_rcv (net/ipv6/xfrm6_input.c:172)
> [...]
>
> Note that one could set CROSS_COMPILE=llvm- instead to hack around this
> issue. However, doing so can break the decodecode routine as it will
> force the selection of other LLVM utilities down the line e.g. llvm-as.
>
> [1] https://lore.kernel.org/all/[email protected]/
>
> Cc: Will Deacon <[email protected]>
> Cc: John Stultz <[email protected]>
> Cc: Masahiro Yamada <[email protected]>
> Signed-off-by: Carlos Llamas <[email protected]>
Thanks for the patch!
Reviewed-by: Nick Desaulniers <[email protected]>
> ---
> scripts/decode_stacktrace.sh | 19 +++++++++++++++++--
> 1 file changed, 17 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/decode_stacktrace.sh b/scripts/decode_stacktrace.sh
> index 564c5632e1a2..bfe5a4082d8e 100755
> --- a/scripts/decode_stacktrace.sh
> +++ b/scripts/decode_stacktrace.sh
> @@ -16,6 +16,21 @@ elif type c++filt >/dev/null 2>&1 ; then
> cppfilt_opts=-i
> fi
>
> +UTIL_SUFFIX=
> +if [[ -z ${LLVM:-} ]]; then
> + UTIL_PREFIX=${CROSS_COMPILE:-}
> +else
> + UTIL_PREFIX=llvm-
> + if [[ ${LLVM} == */ ]]; then
> + UTIL_PREFIX=${LLVM}${UTIL_PREFIX}
> + elif [[ ${LLVM} == -* ]]; then
> + UTIL_SUFFIX=${LLVM}
> + fi
> +fi
> +
> +READELF=${UTIL_PREFIX}readelf${UTIL_SUFFIX}
> +ADDR2LINE=${UTIL_PREFIX}addr2line${UTIL_SUFFIX}
> +
> if [[ $1 == "-r" ]] ; then
> vmlinux=""
> basepath="auto"
> @@ -75,7 +90,7 @@ find_module() {
>
> if [[ "$modpath" != "" ]] ; then
> for fn in $(find "$modpath" -name "${module//_/[-_]}.ko*") ; do
> - if readelf -WS "$fn" | grep -qwF .debug_line ; then
> + if ${READELF} -WS "$fn" | grep -qwF .debug_line ; then
> echo $fn
> return
> fi
> @@ -169,7 +184,7 @@ parse_symbol() {
> if [[ $aarray_support == true && "${cache[$module,$address]+isset}" == "isset" ]]; then
> local code=${cache[$module,$address]}
> else
> - local code=$(${CROSS_COMPILE}addr2line -i -e "$objfile" "$address" 2>/dev/null)
> + local code=$(${ADDR2LINE} -i -e "$objfile" "$address" 2>/dev/null)
> if [[ $aarray_support == true ]]; then
> cache[$module,$address]=$code
> fi
> --
> 2.42.0.582.g8ccd20d70d-goog
>
--
Thanks,
~Nick Desaulniers
On 9/28/2023 8:48 PM, Carlos Llamas wrote:
> GNU's addr2line can have problems parsing a vmlinux built with LLVM,
> particularly when LTO was used. In order to decode the traces correctly
> this patch adds the ability to switch to LLVM's utilities readelf and
> addr2line. The same approach is followed by Will in [1].
>
> Before:
> $ scripts/decode_stacktrace.sh vmlinux < kernel.log
> [17716.240635] Call trace:
> [17716.240646] skb_cow_data (??:?)
> [17716.240654] esp6_input (ld-temp.o:?)
> [17716.240666] xfrm_input (ld-temp.o:?)
> [17716.240674] xfrm6_rcv (??:?)
> [...]
>
> After:
> $ LLVM=1 scripts/decode_stacktrace.sh vmlinux < kernel.log
> [17716.240635] Call trace:
> [17716.240646] skb_cow_data (include/linux/skbuff.h:2172 net/core/skbuff.c:4503)
> [17716.240654] esp6_input (net/ipv6/esp6.c:977)
> [17716.240666] xfrm_input (net/xfrm/xfrm_input.c:659)
> [17716.240674] xfrm6_rcv (net/ipv6/xfrm6_input.c:172)
> [...]
>
> Note that one could set CROSS_COMPILE=llvm- instead to hack around this
> issue. However, doing so can break the decodecode routine as it will
> force the selection of other LLVM utilities down the line e.g. llvm-as.
>
> [1] https://lore.kernel.org/all/[email protected]/
>
> Cc: Will Deacon <[email protected]>
> Cc: John Stultz <[email protected]>
> Cc: Masahiro Yamada <[email protected]>
> Signed-off-by: Carlos Llamas <[email protected]>
Reviewed-by: Elliot Berman <[email protected]>
> ---
> scripts/decode_stacktrace.sh | 19 +++++++++++++++++--
> 1 file changed, 17 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/decode_stacktrace.sh b/scripts/decode_stacktrace.sh
> index 564c5632e1a2..bfe5a4082d8e 100755
> --- a/scripts/decode_stacktrace.sh
> +++ b/scripts/decode_stacktrace.sh
> @@ -16,6 +16,21 @@ elif type c++filt >/dev/null 2>&1 ; then
> cppfilt_opts=-i
> fi
>
> +UTIL_SUFFIX=
> +if [[ -z ${LLVM:-} ]]; then
> + UTIL_PREFIX=${CROSS_COMPILE:-}
> +else
> + UTIL_PREFIX=llvm-
> + if [[ ${LLVM} == */ ]]; then
> + UTIL_PREFIX=${LLVM}${UTIL_PREFIX}
> + elif [[ ${LLVM} == -* ]]; then
> + UTIL_SUFFIX=${LLVM}
> + fi
> +fi
> +
> +READELF=${UTIL_PREFIX}readelf${UTIL_SUFFIX}
> +ADDR2LINE=${UTIL_PREFIX}addr2line${UTIL_SUFFIX}
> +
> if [[ $1 == "-r" ]] ; then
> vmlinux=""
> basepath="auto"
> @@ -75,7 +90,7 @@ find_module() {
>
> if [[ "$modpath" != "" ]] ; then
> for fn in $(find "$modpath" -name "${module//_/[-_]}.ko*") ; do
> - if readelf -WS "$fn" | grep -qwF .debug_line ; then
> + if ${READELF} -WS "$fn" | grep -qwF .debug_line ; then
> echo $fn
> return
> fi
> @@ -169,7 +184,7 @@ parse_symbol() {
> if [[ $aarray_support == true && "${cache[$module,$address]+isset}" == "isset" ]]; then
> local code=${cache[$module,$address]}
> else
> - local code=$(${CROSS_COMPILE}addr2line -i -e "$objfile" "$address" 2>/dev/null)
> + local code=$(${ADDR2LINE} -i -e "$objfile" "$address" 2>/dev/null)
> if [[ $aarray_support == true ]]; then
> cache[$module,$address]=$code
> fi
On Mon, Dec 18, 2023 at 12:08:14PM -0800, Elliot Berman wrote:
>
> Reviewed-by: Elliot Berman <[email protected]>
>
Oh, thanks Elliot for having a look. I had forgotten about this patch.
I'll RESEND if this doesn't get picked up next week.
--
Carlos Llamas
Hi,
On Fri, Sep 29, 2023 at 03:48:17AM +0000, Carlos Llamas wrote:
> GNU's addr2line can have problems parsing a vmlinux built with LLVM,
> particularly when LTO was used. In order to decode the traces correctly
> this patch adds the ability to switch to LLVM's utilities readelf and
> addr2line. The same approach is followed by Will in [1].
>
> Before:
> $ scripts/decode_stacktrace.sh vmlinux < kernel.log
> [17716.240635] Call trace:
> [17716.240646] skb_cow_data (??:?)
> [17716.240654] esp6_input (ld-temp.o:?)
> [17716.240666] xfrm_input (ld-temp.o:?)
> [17716.240674] xfrm6_rcv (??:?)
> [...]
Perhaps it is due to where I am dumping the stack, but I am getting good
stack traces on mainline (with a LLVM-built kernel) without this patch.
With that being said, applying this patch still has good stack traces
and it makes sense if it helps others.
>
> After:
> $ LLVM=1 scripts/decode_stacktrace.sh vmlinux < kernel.log
> [17716.240635] Call trace:
> [17716.240646] skb_cow_data (include/linux/skbuff.h:2172 net/core/skbuff.c:4503)
> [17716.240654] esp6_input (net/ipv6/esp6.c:977)
> [17716.240666] xfrm_input (net/xfrm/xfrm_input.c:659)
> [17716.240674] xfrm6_rcv (net/ipv6/xfrm6_input.c:172)
> [...]
>
> Note that one could set CROSS_COMPILE=llvm- instead to hack around this
> issue. However, doing so can break the decodecode routine as it will
> force the selection of other LLVM utilities down the line e.g. llvm-as.
>
> [1] https://lore.kernel.org/all/[email protected]/
>
> Cc: Will Deacon <[email protected]>
> Cc: John Stultz <[email protected]>
> Cc: Masahiro Yamada <[email protected]>
> Signed-off-by: Carlos Llamas <[email protected]>
Tested-by: Justin Stitt <[email protected]>
Note that this patch is similar: https://lore.kernel.org/all/[email protected]/
> ---
> scripts/decode_stacktrace.sh | 19 +++++++++++++++++--
> 1 file changed, 17 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/decode_stacktrace.sh b/scripts/decode_stacktrace.sh
> index 564c5632e1a2..bfe5a4082d8e 100755
> --- a/scripts/decode_stacktrace.sh
> +++ b/scripts/decode_stacktrace.sh
> @@ -16,6 +16,21 @@ elif type c++filt >/dev/null 2>&1 ; then
> cppfilt_opts=-i
> fi
>
> +UTIL_SUFFIX=
> +if [[ -z ${LLVM:-} ]]; then
> + UTIL_PREFIX=${CROSS_COMPILE:-}
> +else
> + UTIL_PREFIX=llvm-
> + if [[ ${LLVM} == */ ]]; then
> + UTIL_PREFIX=${LLVM}${UTIL_PREFIX}
> + elif [[ ${LLVM} == -* ]]; then
> + UTIL_SUFFIX=${LLVM}
> + fi
> +fi
> +
> +READELF=${UTIL_PREFIX}readelf${UTIL_SUFFIX}
> +ADDR2LINE=${UTIL_PREFIX}addr2line${UTIL_SUFFIX}
> +
> if [[ $1 == "-r" ]] ; then
> vmlinux=""
> basepath="auto"
> @@ -75,7 +90,7 @@ find_module() {
>
> if [[ "$modpath" != "" ]] ; then
> for fn in $(find "$modpath" -name "${module//_/[-_]}.ko*") ; do
> - if readelf -WS "$fn" | grep -qwF .debug_line ; then
> + if ${READELF} -WS "$fn" | grep -qwF .debug_line ; then
> echo $fn
> return
> fi
> @@ -169,7 +184,7 @@ parse_symbol() {
> if [[ $aarray_support == true && "${cache[$module,$address]+isset}" == "isset" ]]; then
> local code=${cache[$module,$address]}
> else
> - local code=$(${CROSS_COMPILE}addr2line -i -e "$objfile" "$address" 2>/dev/null)
> + local code=$(${ADDR2LINE} -i -e "$objfile" "$address" 2>/dev/null)
> if [[ $aarray_support == true ]]; then
> cache[$module,$address]=$code
> fi
> --
> 2.42.0.582.g8ccd20d70d-goog
>
Thanks
Justin
On Tue, Dec 19, 2023 at 12:23:18AM +0000, Justin Stitt wrote:
> Hi,
>
> On Fri, Sep 29, 2023 at 03:48:17AM +0000, Carlos Llamas wrote:
> > GNU's addr2line can have problems parsing a vmlinux built with LLVM,
> > particularly when LTO was used. In order to decode the traces correctly
> > this patch adds the ability to switch to LLVM's utilities readelf and
> > addr2line. The same approach is followed by Will in [1].
> >
> > Before:
> > $ scripts/decode_stacktrace.sh vmlinux < kernel.log
> > [17716.240635] Call trace:
> > [17716.240646] skb_cow_data (??:?)
> > [17716.240654] esp6_input (ld-temp.o:?)
> > [17716.240666] xfrm_input (ld-temp.o:?)
> > [17716.240674] xfrm6_rcv (??:?)
> > [...]
>
> Perhaps it is due to where I am dumping the stack, but I am getting good
> stack traces on mainline (with a LLVM-built kernel) without this patch.
Maybe you have set CROSS_COMPILE=llvm- in your environment? This is an
alternative workaround. I write more about this in the commit log.
>
> With that being said, applying this patch still has good stack traces
> and it makes sense if it helps others.
>
> >
> > After:
> > $ LLVM=1 scripts/decode_stacktrace.sh vmlinux < kernel.log
> > [17716.240635] Call trace:
> > [17716.240646] skb_cow_data (include/linux/skbuff.h:2172 net/core/skbuff.c:4503)
> > [17716.240654] esp6_input (net/ipv6/esp6.c:977)
> > [17716.240666] xfrm_input (net/xfrm/xfrm_input.c:659)
> > [17716.240674] xfrm6_rcv (net/ipv6/xfrm6_input.c:172)
> > [...]
> >
> > Note that one could set CROSS_COMPILE=llvm- instead to hack around this
> > issue. However, doing so can break the decodecode routine as it will
> > force the selection of other LLVM utilities down the line e.g. llvm-as.
> >
> > [1] https://lore.kernel.org/all/[email protected]/
> >
> > Cc: Will Deacon <[email protected]>
> > Cc: John Stultz <[email protected]>
> > Cc: Masahiro Yamada <[email protected]>
> > Signed-off-by: Carlos Llamas <[email protected]>
>
> Tested-by: Justin Stitt <[email protected]>
>
> Note that this patch is similar: https://lore.kernel.org/all/[email protected]/
Ha, I see. Elliot's patch help revive this thread. Cool!
--
Thanks,
Carlos Llamas
On Fri, Sep 29, 2023 at 03:48:17AM +0000, Carlos Llamas wrote:
> GNU's addr2line can have problems parsing a vmlinux built with LLVM,
> particularly when LTO was used. In order to decode the traces correctly
> this patch adds the ability to switch to LLVM's utilities readelf and
> addr2line. The same approach is followed by Will in [1].
>
> Before:
> $ scripts/decode_stacktrace.sh vmlinux < kernel.log
> [17716.240635] Call trace:
> [17716.240646] skb_cow_data (??:?)
> [17716.240654] esp6_input (ld-temp.o:?)
> [17716.240666] xfrm_input (ld-temp.o:?)
> [17716.240674] xfrm6_rcv (??:?)
> [...]
>
> After:
> $ LLVM=1 scripts/decode_stacktrace.sh vmlinux < kernel.log
> [17716.240635] Call trace:
> [17716.240646] skb_cow_data (include/linux/skbuff.h:2172 net/core/skbuff.c:4503)
> [17716.240654] esp6_input (net/ipv6/esp6.c:977)
> [17716.240666] xfrm_input (net/xfrm/xfrm_input.c:659)
> [17716.240674] xfrm6_rcv (net/ipv6/xfrm6_input.c:172)
> [...]
>
> Note that one could set CROSS_COMPILE=llvm- instead to hack around this
> issue. However, doing so can break the decodecode routine as it will
> force the selection of other LLVM utilities down the line e.g. llvm-as.
>
> [1] https://lore.kernel.org/all/[email protected]/
>
> Cc: Will Deacon <[email protected]>
> Cc: John Stultz <[email protected]>
> Cc: Masahiro Yamada <[email protected]>
> Signed-off-by: Carlos Llamas <[email protected]>
> ---
There is no clear maintainer for this script. However, I had a look at
git history and it seems this might need to be picked up by Andrew.
Cc: Andrew Morton <[email protected]>
Andrew can you please pick this up?
--
Carlos Llamas