Some symbols emitted in the readelf output but filtered from System.map
can confuse the 'faddr2line' symbol size calculation, resulting in the
erroneous rejection of valid offsets. This is especially prevalent when
building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions
are prefixed with a 32-bit data value in a '$d.n' section. For example:
447538: ffff800080014b80 548 FUNC GLOBAL DEFAULT 2 do_one_initcall
104: ffff800080014c74 0 NOTYPE LOCAL DEFAULT 2 $x.73
106: ffff800080014d30 0 NOTYPE LOCAL DEFAULT 2 $x.75
111: ffff800080014da4 0 NOTYPE LOCAL DEFAULT 2 $d.78
112: ffff800080014da8 0 NOTYPE LOCAL DEFAULT 2 $x.79
36: ffff800080014de0 200 FUNC LOCAL DEFAULT 2 run_init_process
Adding a warning to do_one_initcall() results in:
| WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260
Which 'faddr2line' refuses to accept:
$ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224)
no match for do_one_initcall+0xf4/0x260
Filter out entries from readelf using the 'sysmap-ignored-syms.sed'
script used to construct System.map, so that the size of a symbol is
calculated as a delta to the next symbol present in ksymtab.
Cc: Josh Poimboeuf <[email protected]>
Cc: John Stultz <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
---
scripts/faddr2line | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/scripts/faddr2line b/scripts/faddr2line
index 62a3fa6f6f59..da734af90036 100755
--- a/scripts/faddr2line
+++ b/scripts/faddr2line
@@ -64,6 +64,7 @@ else
UTIL_PREFIX=${CROSS_COMPILE:-}
fi
+IGNORED_SYMS=$(dirname $0)/sysmap-ignored-syms.sed
READELF="${UTIL_PREFIX}readelf"
ADDR2LINE="${UTIL_PREFIX}addr2line"
AWK="awk"
@@ -185,7 +186,7 @@ __faddr2line() {
found=2
break
fi
- done < <(${READELF} --symbols --wide $objfile | sed 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2)
+ done < <(${READELF} --symbols --wide $objfile | sed -f ${IGNORED_SYMS} -e 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2)
if [[ $found = 0 ]]; then
warn "can't find symbol: sym_name: $sym_name sym_sec: $sym_sec sym_addr: $sym_addr sym_elf_size: $sym_elf_size"
--
2.41.0.487.g6d72f3e995-goog
On Fri, Jul 28, 2023 at 4:34 AM Will Deacon <[email protected]> wrote:
>
> Some symbols emitted in the readelf output but filtered from System.map
> can confuse the 'faddr2line' symbol size calculation, resulting in the
> erroneous rejection of valid offsets. This is especially prevalent when
> building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions
> are prefixed with a 32-bit data value in a '$d.n' section. For example:
>
> 447538: ffff800080014b80 548 FUNC GLOBAL DEFAULT 2 do_one_initcall
> 104: ffff800080014c74 0 NOTYPE LOCAL DEFAULT 2 $x.73
> 106: ffff800080014d30 0 NOTYPE LOCAL DEFAULT 2 $x.75
> 111: ffff800080014da4 0 NOTYPE LOCAL DEFAULT 2 $d.78
> 112: ffff800080014da8 0 NOTYPE LOCAL DEFAULT 2 $x.79
> 36: ffff800080014de0 200 FUNC LOCAL DEFAULT 2 run_init_process
Sami,
Should we change the llvm-ir linkage type for these symbols from
`internal` to `private`?
https://llvm.org/docs/LangRef.html#linkage-types
Then they would not appear in the symbol table.
At first, I thought other modules might need to directly reference
this data, but with the local binding, I don't think they can.
>
> Adding a warning to do_one_initcall() results in:
>
> | WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260
>
> Which 'faddr2line' refuses to accept:
>
> $ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
> skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224)
> no match for do_one_initcall+0xf4/0x260
>
> Filter out entries from readelf using the 'sysmap-ignored-syms.sed'
> script used to construct System.map, so that the size of a symbol is
> calculated as a delta to the next symbol present in ksymtab.
>
> Cc: Josh Poimboeuf <[email protected]>
> Cc: John Stultz <[email protected]>
> Signed-off-by: Will Deacon <[email protected]>
> ---
> scripts/faddr2line | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/faddr2line b/scripts/faddr2line
> index 62a3fa6f6f59..da734af90036 100755
> --- a/scripts/faddr2line
> +++ b/scripts/faddr2line
> @@ -64,6 +64,7 @@ else
> UTIL_PREFIX=${CROSS_COMPILE:-}
> fi
>
> +IGNORED_SYMS=$(dirname $0)/sysmap-ignored-syms.sed
> READELF="${UTIL_PREFIX}readelf"
> ADDR2LINE="${UTIL_PREFIX}addr2line"
> AWK="awk"
> @@ -185,7 +186,7 @@ __faddr2line() {
> found=2
> break
> fi
> - done < <(${READELF} --symbols --wide $objfile | sed 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2)
> + done < <(${READELF} --symbols --wide $objfile | sed -f ${IGNORED_SYMS} -e 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2)
>
> if [[ $found = 0 ]]; then
> warn "can't find symbol: sym_name: $sym_name sym_sec: $sym_sec sym_addr: $sym_addr sym_elf_size: $sym_elf_size"
> --
> 2.41.0.487.g6d72f3e995-goog
>
--
Thanks,
~Nick Desaulniers
On Tue, Aug 1, 2023 at 9:42 AM Nick Desaulniers <[email protected]> wrote:
>
> On Fri, Jul 28, 2023 at 4:34 AM Will Deacon <[email protected]> wrote:
> >
> > Some symbols emitted in the readelf output but filtered from System.map
> > can confuse the 'faddr2line' symbol size calculation, resulting in the
> > erroneous rejection of valid offsets. This is especially prevalent when
> > building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions
> > are prefixed with a 32-bit data value in a '$d.n' section. For example:
> >
> > 447538: ffff800080014b80 548 FUNC GLOBAL DEFAULT 2 do_one_initcall
> > 104: ffff800080014c74 0 NOTYPE LOCAL DEFAULT 2 $x.73
> > 106: ffff800080014d30 0 NOTYPE LOCAL DEFAULT 2 $x.75
> > 111: ffff800080014da4 0 NOTYPE LOCAL DEFAULT 2 $d.78
> > 112: ffff800080014da8 0 NOTYPE LOCAL DEFAULT 2 $x.79
> > 36: ffff800080014de0 200 FUNC LOCAL DEFAULT 2 run_init_process
>
> Sami,
> Should we change the llvm-ir linkage type for these symbols from
> `internal` to `private`?
> https://llvm.org/docs/LangRef.html#linkage-types
>
> Then they would not appear in the symbol table.
>
> At first, I thought other modules might need to directly reference
> this data, but with the local binding, I don't think they can.
For arm64, we don't explicitly emit symbols for type prefixes (see
AsmPrinter::emitKCFITypeId). These mapping symbols are emitted by
AArch64ELFStreamer to mark data and code regions. According to the
AArch64 ELF specification, "All mapping symbols have type STT_NOTYPE
and binding STB_LOCAL," so I assume changing the linkage type isn't an
option:
https://github.com/ARM-software/abi-aa/blob/main/aaelf64/aaelf64.rst#mapping-symbols
Sami
On Fri, Jul 28, 2023 at 8:34 PM Will Deacon <[email protected]> wrote:
>
> Some symbols emitted in the readelf output but filtered from System.map
> can confuse the 'faddr2line' symbol size calculation, resulting in the
> erroneous rejection of valid offsets. This is especially prevalent when
> building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions
> are prefixed with a 32-bit data value in a '$d.n' section. For example:
>
> 447538: ffff800080014b80 548 FUNC GLOBAL DEFAULT 2 do_one_initcall
> 104: ffff800080014c74 0 NOTYPE LOCAL DEFAULT 2 $x.73
> 106: ffff800080014d30 0 NOTYPE LOCAL DEFAULT 2 $x.75
> 111: ffff800080014da4 0 NOTYPE LOCAL DEFAULT 2 $d.78
> 112: ffff800080014da8 0 NOTYPE LOCAL DEFAULT 2 $x.79
> 36: ffff800080014de0 200 FUNC LOCAL DEFAULT 2 run_init_process
>
> Adding a warning to do_one_initcall() results in:
>
> | WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260
>
> Which 'faddr2line' refuses to accept:
>
> $ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
> skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224)
> no match for do_one_initcall+0xf4/0x260
>
> Filter out entries from readelf using the 'sysmap-ignored-syms.sed'
> script used to construct System.map, so that the size of a symbol is
> calculated as a delta to the next symbol present in ksymtab.
I do not think this patch set is the right approach.
I assume faddr2line is meant to work with both vmlinux
and modules.
A problem is that we have different filtering policies wrt kallsyms.
scripts/mksysmap filters symbols in vmlinux,
while kernel/module/kallsyms.c filters ones in modules.
This patch tries to get aligned with the stacktrace of vmlinux,
but that does not seem optimal to the stacktrace of modules.
I have not checked the details, but I guess
the module kallsyms filters less symbols.
https://github.com/torvalds/linux/blob/v6.5-rc4/kernel/module/kallsyms.c#L288
I prefer filtering symbols in the intersection of vmlinux and modules.
is_mapping_symbol() filters symbols you are addressing.
> Cc: Josh Poimboeuf <[email protected]>
> Cc: John Stultz <[email protected]>
> Signed-off-by: Will Deacon <[email protected]>
> ---
> scripts/faddr2line | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/faddr2line b/scripts/faddr2line
> index 62a3fa6f6f59..da734af90036 100755
> --- a/scripts/faddr2line
> +++ b/scripts/faddr2line
> @@ -64,6 +64,7 @@ else
> UTIL_PREFIX=${CROSS_COMPILE:-}
> fi
>
> +IGNORED_SYMS=$(dirname $0)/sysmap-ignored-syms.sed
> READELF="${UTIL_PREFIX}readelf"
> ADDR2LINE="${UTIL_PREFIX}addr2line"
> AWK="awk"
> @@ -185,7 +186,7 @@ __faddr2line() {
> found=2
> break
> fi
> - done < <(${READELF} --symbols --wide $objfile | sed 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2)
> + done < <(${READELF} --symbols --wide $objfile | sed -f ${IGNORED_SYMS} -e 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2)
>
> if [[ $found = 0 ]]; then
> warn "can't find symbol: sym_name: $sym_name sym_sec: $sym_sec sym_addr: $sym_addr sym_elf_size: $sym_elf_size"
> --
> 2.41.0.487.g6d72f3e995-goog
>
--
Best Regards
Masahiro Yamada
On Thu, Aug 03, 2023 at 04:54:37AM +0900, Masahiro Yamada wrote:
> On Fri, Jul 28, 2023 at 8:34 PM Will Deacon <[email protected]> wrote:
> >
> > Some symbols emitted in the readelf output but filtered from System.map
> > can confuse the 'faddr2line' symbol size calculation, resulting in the
> > erroneous rejection of valid offsets. This is especially prevalent when
> > building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions
> > are prefixed with a 32-bit data value in a '$d.n' section. For example:
> >
> > 447538: ffff800080014b80 548 FUNC GLOBAL DEFAULT 2 do_one_initcall
> > 104: ffff800080014c74 0 NOTYPE LOCAL DEFAULT 2 $x.73
> > 106: ffff800080014d30 0 NOTYPE LOCAL DEFAULT 2 $x.75
> > 111: ffff800080014da4 0 NOTYPE LOCAL DEFAULT 2 $d.78
> > 112: ffff800080014da8 0 NOTYPE LOCAL DEFAULT 2 $x.79
> > 36: ffff800080014de0 200 FUNC LOCAL DEFAULT 2 run_init_process
> >
> > Adding a warning to do_one_initcall() results in:
> >
> > | WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260
> >
> > Which 'faddr2line' refuses to accept:
> >
> > $ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
> > skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224)
> > no match for do_one_initcall+0xf4/0x260
> >
> > Filter out entries from readelf using the 'sysmap-ignored-syms.sed'
> > script used to construct System.map, so that the size of a symbol is
> > calculated as a delta to the next symbol present in ksymtab.
>
>
> I do not think this patch set is the right approach.
>
> I assume faddr2line is meant to work with both vmlinux
> and modules.
Huh, it seems to be busted for modules :/ I get:
| error: unknown argument '--section=.text'
with llvm and:
| addr2line: DWARF error: invalid or unhandled FORM value: 0x25
with binutils.
I'll look into this, as I don't think it's related to symbol filtering.
> A problem is that we have different filtering policies wrt kallsyms.
>
> scripts/mksysmap filters symbols in vmlinux,
> while kernel/module/kallsyms.c filters ones in modules.
I don't understand why we need two different ways of filtering out
symbols, but it appears that the module case only filters out local
labels and mapping symbols, both of which are filtered out of vmlinux
as well. Is that right?
> This patch tries to get aligned with the stacktrace of vmlinux,
> but that does not seem optimal to the stacktrace of modules.
>
>
> I have not checked the details, but I guess
> the module kallsyms filters less symbols.
>
> https://github.com/torvalds/linux/blob/v6.5-rc4/kernel/module/kallsyms.c#L288
>
> I prefer filtering symbols in the intersection of vmlinux and modules.
I think mksysmap filters out a superset of the symbols which are filtered
for modules, so why is the intersection the right thing to do? That will
mean that faddr2line considers a whole bunch of symbols that aren't in
the ksymtab of vmlinux.
> is_mapping_symbol() filters symbols you are addressing.
That's a C function and faddr2line is a shell script. What exactly do
you want me to do? My first hack just matched on symbols starting with
'$' but I ended up with this after other review feedback.
Josh -- how do you want to proceed here?
Will
On Fri, Aug 4, 2023 at 11:30 PM Will Deacon <[email protected]> wrote:
>
> On Thu, Aug 03, 2023 at 04:54:37AM +0900, Masahiro Yamada wrote:
> > On Fri, Jul 28, 2023 at 8:34 PM Will Deacon <[email protected]> wrote:
> > >
> > > Some symbols emitted in the readelf output but filtered from System.map
> > > can confuse the 'faddr2line' symbol size calculation, resulting in the
> > > erroneous rejection of valid offsets. This is especially prevalent when
> > > building an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions
> > > are prefixed with a 32-bit data value in a '$d.n' section. For example:
> > >
> > > 447538: ffff800080014b80 548 FUNC GLOBAL DEFAULT 2 do_one_initcall
> > > 104: ffff800080014c74 0 NOTYPE LOCAL DEFAULT 2 $x.73
> > > 106: ffff800080014d30 0 NOTYPE LOCAL DEFAULT 2 $x.75
> > > 111: ffff800080014da4 0 NOTYPE LOCAL DEFAULT 2 $d.78
> > > 112: ffff800080014da8 0 NOTYPE LOCAL DEFAULT 2 $x.79
> > > 36: ffff800080014de0 200 FUNC LOCAL DEFAULT 2 run_init_process
> > >
> > > Adding a warning to do_one_initcall() results in:
> > >
> > > | WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260
> > >
> > > Which 'faddr2line' refuses to accept:
> > >
> > > $ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
> > > skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224)
> > > no match for do_one_initcall+0xf4/0x260
> > >
> > > Filter out entries from readelf using the 'sysmap-ignored-syms.sed'
> > > script used to construct System.map, so that the size of a symbol is
> > > calculated as a delta to the next symbol present in ksymtab.
> >
> >
> > I do not think this patch set is the right approach.
> >
> > I assume faddr2line is meant to work with both vmlinux
> > and modules.
>
> Huh, it seems to be busted for modules :/ I get:
>
> | error: unknown argument '--section=.text'
>
> with llvm and:
>
> | addr2line: DWARF error: invalid or unhandled FORM value: 0x25
>
> with binutils.
>
> I'll look into this, as I don't think it's related to symbol filtering.
>
> > A problem is that we have different filtering policies wrt kallsyms.
> >
> > scripts/mksysmap filters symbols in vmlinux,
> > while kernel/module/kallsyms.c filters ones in modules.
>
> I don't understand why we need two different ways of filtering out
> symbols, but it appears that the module case only filters out local
> labels and mapping symbols, both of which are filtered out of vmlinux
> as well. Is that right?
For vmlinux kallsyms, the reason is the annoyance in the build process.
kallsyms requires linking vmlinux three times.
(see scripts/link-vmlinux.sh if you are interested)
[1a] link vmlinux without symbol table
[1b] Generate symbol list from [1a]
[2a] link vmlinux, embedding the temp kallsyms created by [1b]
[2b] Generate symbol list from [2a]
[3a] link vmlinux, embedding the final kallsyms created by [2b]
Our assumption is that [1b] and [2b] have the same number of
symbols, so that [2a] and [3a] can embed the same size kallsyms.
However, the process [2a] inserts the kallsyms table
in the middle of vmlinux.
It may cause the compiler to emit additional symbols
(e.g. ARM veneers) due to the inserted kallsyms.
So, we filter out all compiler-generated symbols so that
[1b] and [2b] have the same number of symbols.
If we do not do it,
we may have more and more kallsyms steps.
Emitted veneers increase kallsyms size
-> Increased kallsyms causes more veneers
-> The new veneers increase kallsyms size
-> Increased kallsyms causes yet more veneers
-> The new veneers increase kallsyms size again
-> (this cycle may continue again and again)
> > This patch tries to get aligned with the stacktrace of vmlinux,
> > but that does not seem optimal to the stacktrace of modules.
> >
> >
> > I have not checked the details, but I guess
> > the module kallsyms filters less symbols.
> >
> > https://github.com/torvalds/linux/blob/v6.5-rc4/kernel/module/kallsyms.c#L288
> >
> > I prefer filtering symbols in the intersection of vmlinux and modules.
>
> I think mksysmap filters out a superset of the symbols which are filtered
> for modules, so why is the intersection the right thing to do? That will
> mean that faddr2line considers a whole bunch of symbols that aren't in
> the ksymtab of vmlinux.
If A is a subset of B,
(A & B) == A.
I meant "the intersection of the filtering rules of vmlinux and modules"
== "the filtering rule of modules".
>
> > is_mapping_symbol() filters symbols you are addressing.
>
> That's a C function and faddr2line is a shell script. What exactly do
> you want me to do? My first hack just matched on symbols starting with
> '$' but I ended up with this after other review feedback.
In linux-next, I see this:
static inline int is_mapping_symbol(const char *str)
{
if (str[0] == '.' && str[1] == 'L')
return true;
if (str[0] == 'L' && str[1] == '0')
return true;
return str[0] == '$';
}
It is pretty easy to write a sed script to do the same thing.
Another possibility is to use different filtering rules
between vmlinux and modules.
(but I do not know if this complexity is worthwhile)
[pseudo code]
get_symbol_list () {
if [ "${is_vmlinux}" = 1 ]; then
{READELF} --symbols --wide $objfile | sed -f
scripts/sysmap-ignored-syms.sed
else
{READELF} --symbols --wide $objfile | sed 'remove ".L", 'L0", "^$"'
fi
}
>
> Josh -- how do you want to proceed here?
>
> Will
--
Best Regards
Masahiro Yamada