Followup to: <[email protected]>
By author: linux-os <[email protected]>
In newsgroup: linux.dev.kernel
> >
> > Actually -mregparm=0 is not supposed to be even accepted by x86-64
> > compiler (I've disabled the function attribute but apparently missed
> > this one) and even if GCC produced valid code by miracle, you will get
> > into trouble with hand written assembly.
>
> Huh? That's the default for a 'C' compiler (not to pass parameters
> in registers). The parameters are passed on the stack as the default!
> The return values don't count. They are, by default passed in eax
> or edx-eax pair for a long long.
>
Dear Wrongbot,
It depends on the architecture ABI. This is the case for the i386
ABI, but definitely *NOT* for x86-64.
-hpa
On Sat, 15 Jan 2005, H. Peter Anvin wrote:
> It depends on the architecture ABI. This is the case for the i386
> ABI, but definitely *NOT* for x86-64.
<begin stuff superseded by later findings, but left here to clarify the
train of thoughts>
Yes, precisely. The ABI/x86_64 defines this behaviour explicitly. However,
that would mean the ABI was designed without giving thought to debugging
via kdb.
When I said "2-3 weeks of work" I didn't fully realize the complexity of
the problem. It is actually more like several months of research work and
then (most likely) coming to the conclusion that the code to simulate the
cpu (by disassembling the functions to track down where those registers
went in each function) is just too complex to be written.
So, this means there is no way for kdb on x86_64 to show the parameter
values for each function in the back trace. Any chance of changing the
ABI/x86_64 to do the right (i.e. passing via stack like on i386) thing
now? Then kdb would automatically support it via normal ar-handling code.
Also, we should NOT claim that Linux has been ported to x86_64
architecture yet, because the port is not clean, i.e. doesn't allow
breaking ABI like it allows on i386, by means of CONFIG_REGPARM
optimization. Anyone working on fixing this?
I cc'd Linus as I cannot believe he agreed with allowing such an
optimization to be a default and standard thing accepted by the Linux
kernel. (But I may be wrong, especially since Linus isn't particularly
fond of kdb anyway :)
<end of stuff declared as superceded>
Actually, having cc'd Linus made me think very _carefully_ about what I
say and I went and checked how the userspace does it, as I couldn't
believe that such fine piece of software as gdb would be broken as well.
And to my surprize I discovered that gdb (when a program is compiled with
-g) works fine! I.e. it shows the function arguments correctly. And
disassembling the functions shows that although the arguments are passed
via registers (as ABI demands) they are also saved somewhere on the stack.
Hmmm, interesting, then -g compiled Linux kernel should also be useable,
with perhaps some tweaks to kdb to decode these frames correctly, right?
I just want someone who knows the answer to confirm the last statement and
so then I can go ahead and (try to) implement the solution by mimicking
gdb behaviour (or help Jack Vogel implement it if he prefers doing it
himself as defacto kdb/x86_64 maintainer)
Kind regards
Tigran
On Mon, 2005-01-17 at 09:30 +0000, Tigran Aivazian wrote:
> I cc'd Linus as I cannot believe he agreed with allowing such an
> optimization to be a default and standard thing accepted by the Linux
> kernel. (But I may be wrong, especially since Linus isn't particularly
> fond of kdb anyway :)
I don't see a problem, have you ever seen ia64??
> Actually, having cc'd Linus made me think very _carefully_ about what I
> say and I went and checked how the userspace does it, as I couldn't
> believe that such fine piece of software as gdb would be broken as well.
> And to my surprize I discovered that gdb (when a program is compiled with
> -g) works fine! I.e. it shows the function arguments correctly. And
so why don't you use kgdb instead of kdb ?
On Mon, 17 Jan 2005, Arjan van de Ven wrote:
>> Actually, having cc'd Linus made me think very _carefully_ about what I
>> say and I went and checked how the userspace does it, as I couldn't
>> believe that such fine piece of software as gdb would be broken as well.
>> And to my surprize I discovered that gdb (when a program is compiled with
>> -g) works fine! I.e. it shows the function arguments correctly. And
>
>
> so why don't you use kgdb instead of kdb ?
If kdb was some dead unmaintained piece of software then, yes, I would
follow your advice and switch to kgdb. But kdb is a very nice and actively
maintained piece of work, so it should be fixed to show the parameter
values correctly in the backtrace.
Kind regards
Tigran
Tigran Aivazian <[email protected]> writes:
>
> When I said "2-3 weeks of work" I didn't fully realize the complexity
> of the problem. It is actually more like several months of research
> work and then (most likely) coming to the conclusion that the code to
> simulate the cpu (by disassembling the functions to track down where
> those registers went in each function) is just too complex to be
> written.
Did you actually ever read the ABI?
The ABI supported way is to read the DWARF2 unwind tables. For that
you would a dwarf2 reader. gdb does that in user space, and libgcc2
also does it for exception unwinding. IA64 has an in kernel dwarf2
reader library (and ia64 kdb uses it), although it would probably need
some work to make it work on x86-64.
So far nobody wanted it enough to do the porting work though.
-Andi
Hi Andi,
On Mon, 17 Jan 2005, Andi Kleen wrote:
> The ABI supported way is to read the DWARF2 unwind tables. For that
> you would a dwarf2 reader. gdb does that in user space, and libgcc2
> also does it for exception unwinding. IA64 has an in kernel dwarf2
> reader library (and ia64 kdb uses it), although it would probably need
> some work to make it work on x86-64.
>
> So far nobody wanted it enough to do the porting work though.
Thank you for the pointer. I will look at it and see if I can try to port
it to x86_64, since that is the standard and official way. I admit that I
didn't realize that the "magic stuff" which gdb does is in fact a dwarf2
implementation (and this is what's missing in kdb on x86_64).
Kind regards
Tigran
On Mon, 17 Jan 2005, Tigran Aivazian wrote:
>
> So, this means there is no way for kdb on x86_64 to show the parameter
> values for each function in the back trace. Any chance of changing the
> ABI/x86_64 to do the right (i.e. passing via stack like on i386) thing
> now? Then kdb would automatically support it via normal ar-handling code.
"right thing"? You have a very strange definition of "right". The x86 is
pretty much the only architecture still in use that passes everything on
the stack by default, and even there it's considered pretty painful (and
is not true of FP arguments).
If a debugger cannot handle arguments in registers, it by definition
cannot handle things like alpha/ppc/mips/xxx, so I'd say that the
debugger is seriously broken.
Linus
Tigran Aivazian wrote:
> On Mon, 17 Jan 2005, Arjan van de Ven wrote:
>
>>> Actually, having cc'd Linus made me think very _carefully_ about what I
>>> say and I went and checked how the userspace does it, as I couldn't
>>> believe that such fine piece of software as gdb would be broken as well.
>>> And to my surprize I discovered that gdb (when a program is compiled
>>> with
>>> -g) works fine! I.e. it shows the function arguments correctly. And
>>
>> so why don't you use kgdb instead of kdb ?
>
> If kdb was some dead unmaintained piece of software then, yes, I would
> follow your advice and switch to kgdb. But kdb is a very nice and
> actively maintained piece of work, so it should be fixed to show the
> parameter values correctly in the backtrace.
That's a kdb maintainer issue. The x86-64 folks have nicely provided a
set of libraries to do backtraces, etc. Your previous rant is just so
far off base it's not even funny.
-hpa
"H. Peter Anvin" <[email protected]> writes:
> Tigran Aivazian wrote:
>> On Mon, 17 Jan 2005, Arjan van de Ven wrote:
>>
>>>> Actually, having cc'd Linus made me think very _carefully_ about what I
>>>> say and I went and checked how the userspace does it, as I couldn't
>>>> believe that such fine piece of software as gdb would be broken as well.
>>>> And to my surprize I discovered that gdb (when a program is
>>>> compiled with
>>>> -g) works fine! I.e. it shows the function arguments correctly. And
>>>
>>> so why don't you use kgdb instead of kdb ?
>> If kdb was some dead unmaintained piece of software then, yes, I
>> would follow your advice and switch to kgdb. But kdb is a very nice
>> and actively maintained piece of work, so it should be fixed to show
>> the parameter values correctly in the backtrace.
>
> That's a kdb maintainer issue. The x86-64 folks have nicely provided
> a set of libraries to do backtraces, etc. Your previous rant is just
> so far off base it's not even funny.
To be fair there isn't a nice library for it on x86-64. There
is libunwind on IA64, but afaik nobody ported it to x86-64 yet.
Just various projects have their own private unwind
implementation. The kernel including KDB has always lived with
imprecise backtraces and no argument printing. I don't think it has
been a show stopper so far. If you really want the arguments you can
always use kgdb.
However I'm not sure we really want libunwind in the kernel anyways
(not even in KDB ;-) If anything better something stripped down and
simple which libunwind isn't.
Unfortunately dwarf2 is not exactly a simple spec so implementing
a new backtracer for the kernel is not a trivial task.
-Andi
Andi Kleen wrote:
>
> To be fair there isn't a nice library for it on x86-64. There
> is libunwind on IA64, but afaik nobody ported it to x86-64 yet.
>
> Just various projects have their own private unwind
> implementation. The kernel including KDB has always lived with
> imprecise backtraces and no argument printing. I don't think it has
> been a show stopper so far. If you really want the arguments you can
> always use kgdb.
>
> However I'm not sure we really want libunwind in the kernel anyways
> (not even in KDB ;-) If anything better something stripped down and
> simple which libunwind isn't.
>
> Unfortunately dwarf2 is not exactly a simple spec so implementing
> a new backtracer for the kernel is not a trivial task.
>
Seems like the unwinder should be running client-side, like it does on
kgdb. Or does kdb not have a client at all? (If so, I have no sympathy
for it.)
-hpa
> Seems like the unwinder should be running client-side, like it does on
> kgdb. Or does kdb not have a client at all? (If so, I have no sympathy
> for it.)
kdb runs 100% in the kernel.
-Andi
On Mon, 17 Jan 2005, H. Peter Anvin wrote:
> Or does kdb not have a client at all? (If so, I have no sympathy for it.)
Peter, it was very easy for you to call my emails "rants" and "not even
funny" but the above statement is displaying complete ignorance of what
kdb actually is :) So, instead of "patronizing" your fellow-hacker, please
listen to what he has to say below:
I already solved this paricular problem. And the solution is (but don't
tell me you knew it, for then why didn't you tell anyone) simply ---
compile the kernel with -g and that includes enough debug information to
be able to decode the stack content correctly. And yes, kdb does show the
correct argument values now. No changes to kdb are necessary and no need
to do the work with dwarf2 implementation etc etc.
However, this highlighted a more serious problem in the x86_64 kernel (or
more likely in the kdb patch) --- the kernel compiled with -g panics when
you try to return from kdb after hitting a breakpoint. This is a bug and
I'll investigate to find out the reason why it panics. (I hope it is not
an "assumption" of the x86_64 port that one must never compile the kernel
with -g either...)
Kind regards
Tigran
On Tue, 2005-01-18 at 11:25 +0000, Tigran Aivazian wrote:
> However, this highlighted a more serious problem in the x86_64 kernel (or
> more likely in the kdb patch) --- the kernel compiled with -g panics when
> you try to return from kdb after hitting a breakpoint. This is a bug and
> I'll investigate to find out the reason why it panics. (I hope it is not
> an "assumption" of the x86_64 port that one must never compile the kernel
> with -g either...)
That is isn't; for example the Fedora Core 2 and 3 and the Red Hat
Enterprise Linux 4 kernels are compiled wit -g.
On Tue, 18 Jan 2005, Tigran Aivazian wrote:
> I already solved this paricular problem. And the solution is (but don't tell
> me you knew it, for then why didn't you tell anyone) simply --- compile the
> kernel with -g and that includes enough debug information to be able to
> decode the stack content correctly. And yes, kdb does show the correct
> argument values now. No changes to kdb are necessary and no need to do the
> work with dwarf2 implementation etc etc.
actually I am very surprized that it worked (because looking at the code I
concluded that it should NOT). So I need to retest in all cases to make
sure this is not a coincidence but a solid fact...
Kind regards
Tigran
Sorry, it was a "red herring", namely my module's Makefile still had
"-mregparm=0" in CFLAGS, so that is why kdb showed it's arguments
correctly (and then paniced on the way out).
So we still have to deal with DWARF2 data then... Ok, will look into it.
Kind regards
Tigran
On Tue, 18 Jan 2005, Tigran Aivazian wrote:
> On Tue, 18 Jan 2005, Tigran Aivazian wrote:
>> I already solved this paricular problem. And the solution is (but don't
>> tell me you knew it, for then why didn't you tell anyone) simply ---
>> compile the kernel with -g and that includes enough debug information to be
>> able to decode the stack content correctly. And yes, kdb does show the
>> correct argument values now. No changes to kdb are necessary and no need to
>> do the work with dwarf2 implementation etc etc.
>
> actually I am very surprized that it worked (because looking at the code I
> concluded that it should NOT). So I need to retest in all cases to make sure
> this is not a coincidence but a solid fact...
>
> Kind regards
> Tigran
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
On Mon, 17 Jan 2005 09:30:17 +0000 (GMT),
Tigran Aivazian <[email protected]> wrote:
>Hmmm, interesting, then -g compiled Linux kernel should also be useable,
>with perhaps some tweaks to kdb to decode these frames correctly, right?
kdb on i386 uses heuristics to guess at what parameters have been
passed on stack. When the parameters are passed by register, there is
not enough information in the code to work out which parameters have
been passed nor what gcc has done with them once the function has been
entered.
Why use heuristics and guess? Because when kdb was started we were
still using a.out and stabs. In those days there was no way for code
in the running kernel to access the kernel's debugging information to
track the parameter and register usage.
Since then the world has moved on. The IA64 ABI mandates that the
information required to do backtrace is stored in the running kernel,
and kdb uses that unwind data, including tracking parameters passed in
registers.
Nobody has been concerned enough about the backtraces on i386 and
x86_64 to add the required unwind data to the kernel for those
platforms. If you want to extract the dwarf data from a kernel
compiled with -g, include the dwarf data in the running kernel and add
a dwarf unwinder to the kernel then I will happily accept patches to
kdb. Don't forget about support for adding and removing unwind data as
modules are loaded and unloaded.
To hpa: kdb is designed to run completely in kernel. It is not (repeat
not) intended to be a remote client debugger.
BTW, even on IA64 which has unwind data, we still get problems because
the unwind data only says what parameters are passed in registers, it
says nothing about register reuse. gcc can reuse a parameter register
if the parameter value is no longer required, for example :-
void function foo(struct bar *b)
{
// On entry, ia64 r32 contains *b
int i = b->i;
// If b is not used after this point, gcc can reuse r32 for the value of i.
// Debugging after this point will show a misleading value in r32 for b.
...
}
Keith Owens <[email protected]> writes:
> Nobody has been concerned enough about the backtraces on i386 and
> x86_64 to add the required unwind data to the kernel for those
> platforms. If you want to extract the dwarf data from a kernel
> compiled with -g, include the dwarf data in the running kernel and add
> a dwarf unwinder to the kernel then I will happily accept patches to
> kdb. Don't forget about support for adding and removing unwind data as
It would be pretty easy to do.
The x86-64 ABI actually includes unwind data (without other dwarf
data) by default in all executables. However it wasn't needed in the
kernel so far so I turned it off to save some disk space:
If you want it without -g just remove the
ifneq ($(CONFIG_DEBUG_INFO),y)
CFLAGS += -fno-asynchronous-unwind-tables
endif
in arch/x86_64/Makefile. Then to actually use it in the running kernel
you would need to change the unwind segment in the vmlinux.lds.S
to be loaded instead of discarded at link time (one liner change too)
And something to map it for modules (i haven't looked at that, but
I suppose if ia64 has the infrastructure it shouldn't be hard to port)
I wouldn't be opposed to a new CONFIG_RUNTIME_UNWIND that does all
this. However without an working unwinder in kernel it's not very useful.
>
> BTW, even on IA64 which has unwind data, we still get problems because
> the unwind data only says what parameters are passed in registers, it
> says nothing about register reuse. gcc can reuse a parameter register
> if the parameter value is no longer required, for example :-
This is no different from stack based parameters where the stack slot
of the parameter can be overwritten by the callee too.
You just will have to live with that.
-Andi