Hi,
When I use kprobe to do something, I found some wired thing.
When CONFIG_FUNCTION_TRACER is disabled:
(gdb) disassemble do_fork
Dump of assembler code for function do_fork:
0xc0037390 <+0>: mflr r0
0xc0037394 <+4>: stwu r1,-64(r1)
0xc0037398 <+8>: mfcr r12
0xc003739c <+12>: stmw r27,44(r1)
Then I:
modprobe kprobe_example func=do_fork offset=4
ls
Things works well.
But when CONFIG_FUNCTION_TRACER is enabled:
(gdb) disassemble do_fork
Dump of assembler code for function do_fork:
0xc0040334 <+0>: mflr r0
0xc0040338 <+4>: stw r0,4(r1)
0xc004033c <+8>: bl 0xc00109d4 <mcount>
0xc0040340 <+12>: stwu r1,-80(r1)
0xc0040344 <+16>: mflr r0
0xc0040348 <+20>: stw r0,84(r1)
0xc004034c <+24>: mfcr r12
Then I:
modprobe kprobe_example func=do_fork offset=12
ls
'ls' will never retrun. system freeze.
I'm using toolchain from:http://www.denx.de/wiki/ELDK-5/WebHome
powerpc-linux-gcc -v
Using built-in specs.
COLLECT_GCC=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/bin/powerpc-linux/powerpc-linux-gcc
COLLECT_LTO_WRAPPER=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/libexec/powerpc-linux/gcc/powerpc-linux/4.5.1/lto-wrapper
Target: powerpc-linux
Configured with:
/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/work/i686-nativesdk-oesdk-linux/gcc-cross-canadian-powerpc-4.5.1-r4/gcc-4.5.1/configure
--build=x86_64-linux --host=i686-oesdk-linux --target=powerpc-linux
--prefix=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr
--exec_prefix=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr
--bindir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/bin/powerpc-linux
--sbindir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/bin/powerpc-linux
--libexecdir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/libexec/powerpc-linux
--datadir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/share
--sysconfdir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/etc
--sharedstatedir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/com
--localstatedir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/var
--libdir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/lib/powerpc-linux
--includedir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/include
--oldincludedir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/include
--infodir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/share/info
--mandir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/share/man
--with-libtool-sysroot=/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/sysroots/i686-oesdk-linux-nativesdk
--with-gnu-ld --enable-shared --enable-languages=c,c++
--enable-threads=posix --disable-multilib --enable-c99
--enable-long-long --enable-symvers=gnu --enable-libstdcxx-pch
--program-prefix=powerpc-linux- --enable-lto --enable-libssp
--disable-bootstrap --disable-libgomp --disable-libmudflap
--enable-cheaders=c_global
--with-local-prefix=/opt/eldk-5.0/powerpc/sysroots/powerpc-linux/usr
--with-gxx-include-dir=/usr/include/c++
--with-build-time-tools=/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/sysroots/x86_64-linux/usr/powerpc-linux/bin
--with-sysroot=/opt/eldk-5.0/powerpc/sysroots/powerpc-linux
--with-build-sysroot=/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/sysroots/powerpc
--disable-libunwind-exceptions --disable-libssp --disable-libgomp
--disable-libmudflap
--with-mpfr=/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/sysroots/i686-oesdk-linux-nativesdk
--with-mpc=/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/sysroots/i686-oesdk-linux-nativesdk
--enable-__cxa_atexit
Thread model: posix
gcc version 4.5.1 (GCC)
And kernel config is attached.
BTW, I have made a patch to make kprobe_example set breakpoint easily,
attached too.
Thanks,
Yong
--
Only stand for myself
On Fri, 2011-06-24 at 17:21 +0800, Yong Zhang wrote:
> Hi,
>
> When I use kprobe to do something, I found some wired thing.
>
> When CONFIG_FUNCTION_TRACER is disabled:
> (gdb) disassemble do_fork
> Dump of assembler code for function do_fork:
> 0xc0037390 <+0>: mflr r0
> 0xc0037394 <+4>: stwu r1,-64(r1)
> 0xc0037398 <+8>: mfcr r12
> 0xc003739c <+12>: stmw r27,44(r1)
>
> Then I:
> modprobe kprobe_example func=do_fork offset=4
> ls
> Things works well.
>
> But when CONFIG_FUNCTION_TRACER is enabled:
> (gdb) disassemble do_fork
> Dump of assembler code for function do_fork:
> 0xc0040334 <+0>: mflr r0
> 0xc0040338 <+4>: stw r0,4(r1)
> 0xc004033c <+8>: bl 0xc00109d4 <mcount>
> 0xc0040340 <+12>: stwu r1,-80(r1)
> 0xc0040344 <+16>: mflr r0
> 0xc0040348 <+20>: stw r0,84(r1)
> 0xc004034c <+24>: mfcr r12
> Then I:
> modprobe kprobe_example func=do_fork offset=12
> ls
> 'ls' will never retrun. system freeze.
I'm not sure if x86 had a similar issue.
Masami, have any ideas to why this happened?
I don't have a PPC32 to test on, but I can try it out on my PPC64.
-- Steve
>
> I'm using toolchain from:http://www.denx.de/wiki/ELDK-5/WebHome
> powerpc-linux-gcc -v
> Using built-in specs.
> COLLECT_GCC=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/bin/powerpc-linux/powerpc-linux-gcc
> COLLECT_LTO_WRAPPER=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/libexec/powerpc-linux/gcc/powerpc-linux/4.5.1/lto-wrapper
> Target: powerpc-linux
> Configured with:
> /opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/work/i686-nativesdk-oesdk-linux/gcc-cross-canadian-powerpc-4.5.1-r4/gcc-4.5.1/configure
> --build=x86_64-linux --host=i686-oesdk-linux --target=powerpc-linux
> --prefix=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr
> --exec_prefix=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr
> --bindir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/bin/powerpc-linux
> --sbindir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/bin/powerpc-linux
> --libexecdir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/libexec/powerpc-linux
> --datadir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/share
> --sysconfdir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/etc
> --sharedstatedir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/com
> --localstatedir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/var
> --libdir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/lib/powerpc-linux
> --includedir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/include
> --oldincludedir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/include
> --infodir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/share/info
> --mandir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/share/man
> --with-libtool-sysroot=/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/sysroots/i686-oesdk-linux-nativesdk
> --with-gnu-ld --enable-shared --enable-languages=c,c++
> --enable-threads=posix --disable-multilib --enable-c99
> --enable-long-long --enable-symvers=gnu --enable-libstdcxx-pch
> --program-prefix=powerpc-linux- --enable-lto --enable-libssp
> --disable-bootstrap --disable-libgomp --disable-libmudflap
> --enable-cheaders=c_global
> --with-local-prefix=/opt/eldk-5.0/powerpc/sysroots/powerpc-linux/usr
> --with-gxx-include-dir=/usr/include/c++
> --with-build-time-tools=/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/sysroots/x86_64-linux/usr/powerpc-linux/bin
> --with-sysroot=/opt/eldk-5.0/powerpc/sysroots/powerpc-linux
> --with-build-sysroot=/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/sysroots/powerpc
> --disable-libunwind-exceptions --disable-libssp --disable-libgomp
> --disable-libmudflap
> --with-mpfr=/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/sysroots/i686-oesdk-linux-nativesdk
> --with-mpc=/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/sysroots/i686-oesdk-linux-nativesdk
> --enable-__cxa_atexit
> Thread model: posix
> gcc version 4.5.1 (GCC)
>
>
> And kernel config is attached.
>
> BTW, I have made a patch to make kprobe_example set breakpoint easily,
> attached too.
>
> Thanks,
> Yong
>
(2011/06/24 19:29), Steven Rostedt wrote:
> On Fri, 2011-06-24 at 17:21 +0800, Yong Zhang wrote:
>> Hi,
>>
>> When I use kprobe to do something, I found some wired thing.
>>
>> When CONFIG_FUNCTION_TRACER is disabled:
>> (gdb) disassemble do_fork
>> Dump of assembler code for function do_fork:
>> 0xc0037390 <+0>: mflr r0
>> 0xc0037394 <+4>: stwu r1,-64(r1)
>> 0xc0037398 <+8>: mfcr r12
>> 0xc003739c <+12>: stmw r27,44(r1)
>>
>> Then I:
>> modprobe kprobe_example func=do_fork offset=4
>> ls
>> Things works well.
>>
>> But when CONFIG_FUNCTION_TRACER is enabled:
>> (gdb) disassemble do_fork
>> Dump of assembler code for function do_fork:
>> 0xc0040334 <+0>: mflr r0
>> 0xc0040338 <+4>: stw r0,4(r1)
>> 0xc004033c <+8>: bl 0xc00109d4 <mcount>
>> 0xc0040340 <+12>: stwu r1,-80(r1)
>> 0xc0040344 <+16>: mflr r0
>> 0xc0040348 <+20>: stw r0,84(r1)
>> 0xc004034c <+24>: mfcr r12
>> Then I:
>> modprobe kprobe_example func=do_fork offset=12
>> ls
>> 'ls' will never retrun. system freeze.
>
> I'm not sure if x86 had a similar issue.
>
> Masami, have any ideas to why this happened?
No, I don't familiar with ppc implementation. I guess
that single-step resume code failed to emulate the
instruction, but it strongly depends on ppc arch.
Maybe IBM people may know what happened.
Ananth, Jim, would you have any ideas?
Thank you,
>
> I don't have a PPC32 to test on, but I can try it out on my PPC64.
>
> -- Steve
>
>>
>> I'm using toolchain from:http://www.denx.de/wiki/ELDK-5/WebHome
>> powerpc-linux-gcc -v
>> Using built-in specs.
>> COLLECT_GCC=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/bin/powerpc-linux/powerpc-linux-gcc
>> COLLECT_LTO_WRAPPER=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/libexec/powerpc-linux/gcc/powerpc-linux/4.5.1/lto-wrapper
>> Target: powerpc-linux
>> Configured with:
>> /opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/work/i686-nativesdk-oesdk-linux/gcc-cross-canadian-powerpc-4.5.1-r4/gcc-4.5.1/configure
>> --build=x86_64-linux --host=i686-oesdk-linux --target=powerpc-linux
>> --prefix=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr
>> --exec_prefix=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr
>> --bindir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/bin/powerpc-linux
>> --sbindir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/bin/powerpc-linux
>> --libexecdir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/libexec/powerpc-linux
>> --datadir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/share
>> --sysconfdir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/etc
>> --sharedstatedir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/com
>> --localstatedir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/var
>> --libdir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/lib/powerpc-linux
>> --includedir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/include
>> --oldincludedir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/include
>> --infodir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/share/info
>> --mandir=/opt/eldk-5.0/powerpc/sysroots/i686-oesdk-linux/usr/share/man
>> --with-libtool-sysroot=/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/sysroots/i686-oesdk-linux-nativesdk
>> --with-gnu-ld --enable-shared --enable-languages=c,c++
>> --enable-threads=posix --disable-multilib --enable-c99
>> --enable-long-long --enable-symvers=gnu --enable-libstdcxx-pch
>> --program-prefix=powerpc-linux- --enable-lto --enable-libssp
>> --disable-bootstrap --disable-libgomp --disable-libmudflap
>> --enable-cheaders=c_global
>> --with-local-prefix=/opt/eldk-5.0/powerpc/sysroots/powerpc-linux/usr
>> --with-gxx-include-dir=/usr/include/c++
>> --with-build-time-tools=/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/sysroots/x86_64-linux/usr/powerpc-linux/bin
>> --with-sysroot=/opt/eldk-5.0/powerpc/sysroots/powerpc-linux
>> --with-build-sysroot=/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/sysroots/powerpc
>> --disable-libunwind-exceptions --disable-libssp --disable-libgomp
>> --disable-libmudflap
>> --with-mpfr=/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/sysroots/i686-oesdk-linux-nativesdk
>> --with-mpc=/opt/poky/build/eldk-2011-05-20-5cde06e-powerpc/tmp/sysroots/i686-oesdk-linux-nativesdk
>> --enable-__cxa_atexit
>> Thread model: posix
>> gcc version 4.5.1 (GCC)
>>
>>
>> And kernel config is attached.
>>
>> BTW, I have made a patch to make kprobe_example set breakpoint easily,
>> attached too.
>>
>> Thanks,
>> Yong
>>
--
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
On Sun, Jun 26, 2011 at 11:47:13PM +0900, Masami Hiramatsu wrote:
> (2011/06/24 19:29), Steven Rostedt wrote:
> > On Fri, 2011-06-24 at 17:21 +0800, Yong Zhang wrote:
> >> Hi,
> >>
> >> When I use kprobe to do something, I found some wired thing.
> >>
> >> When CONFIG_FUNCTION_TRACER is disabled:
> >> (gdb) disassemble do_fork
> >> Dump of assembler code for function do_fork:
> >> 0xc0037390 <+0>: mflr r0
> >> 0xc0037394 <+4>: stwu r1,-64(r1)
> >> 0xc0037398 <+8>: mfcr r12
> >> 0xc003739c <+12>: stmw r27,44(r1)
> >>
> >> Then I:
> >> modprobe kprobe_example func=do_fork offset=4
> >> ls
> >> Things works well.
> >>
> >> But when CONFIG_FUNCTION_TRACER is enabled:
> >> (gdb) disassemble do_fork
> >> Dump of assembler code for function do_fork:
> >> 0xc0040334 <+0>: mflr r0
> >> 0xc0040338 <+4>: stw r0,4(r1)
> >> 0xc004033c <+8>: bl 0xc00109d4 <mcount>
> >> 0xc0040340 <+12>: stwu r1,-80(r1)
> >> 0xc0040344 <+16>: mflr r0
> >> 0xc0040348 <+20>: stw r0,84(r1)
> >> 0xc004034c <+24>: mfcr r12
> >> Then I:
> >> modprobe kprobe_example func=do_fork offset=12
> >> ls
> >> 'ls' will never retrun. system freeze.
> >
> > I'm not sure if x86 had a similar issue.
> >
> > Masami, have any ideas to why this happened?
>
> No, I don't familiar with ppc implementation. I guess
> that single-step resume code failed to emulate the
> instruction, but it strongly depends on ppc arch.
> Maybe IBM people may know what happened.
>
> Ananth, Jim, would you have any ideas?
On powerpc, we emulate sstep whenever possible. Only recently support to
emulate loads and stores got added. I don't have access to a powerpc box
today... but will try to recreate the problem ASAP and see what could be
happening in the presence of mcount.
Ananth
On Mon, Jun 27, 2011 at 03:31:05PM +0530, Ananth N Mavinakayanahalli wrote:
> On Sun, Jun 26, 2011 at 11:47:13PM +0900, Masami Hiramatsu wrote:
> > (2011/06/24 19:29), Steven Rostedt wrote:
> > > On Fri, 2011-06-24 at 17:21 +0800, Yong Zhang wrote:
> > >> Hi,
> > >>
> > >> When I use kprobe to do something, I found some wired thing.
> > >>
> > >> When CONFIG_FUNCTION_TRACER is disabled:
> > >> (gdb) disassemble do_fork
> > >> Dump of assembler code for function do_fork:
> > >> 0xc0037390 <+0>: mflr r0
> > >> 0xc0037394 <+4>: stwu r1,-64(r1)
> > >> 0xc0037398 <+8>: mfcr r12
> > >> 0xc003739c <+12>: stmw r27,44(r1)
> > >>
> > >> Then I:
> > >> modprobe kprobe_example func=do_fork offset=4
> > >> ls
> > >> Things works well.
> > >>
> > >> But when CONFIG_FUNCTION_TRACER is enabled:
> > >> (gdb) disassemble do_fork
> > >> Dump of assembler code for function do_fork:
> > >> 0xc0040334 <+0>: mflr r0
> > >> 0xc0040338 <+4>: stw r0,4(r1)
> > >> 0xc004033c <+8>: bl 0xc00109d4 <mcount>
> > >> 0xc0040340 <+12>: stwu r1,-80(r1)
> > >> 0xc0040344 <+16>: mflr r0
> > >> 0xc0040348 <+20>: stw r0,84(r1)
> > >> 0xc004034c <+24>: mfcr r12
> > >> Then I:
> > >> modprobe kprobe_example func=do_fork offset=12
> > >> ls
> > >> 'ls' will never retrun. system freeze.
My access to a 32bit powerpc box is very limited. Also, embedded powerpc
has had issues with gcc-4.6 while gcc-4.5 worked fine.
> > > I'm not sure if x86 had a similar issue.
> > >
> > > Masami, have any ideas to why this happened?
> >
> > No, I don't familiar with ppc implementation. I guess
> > that single-step resume code failed to emulate the
> > instruction, but it strongly depends on ppc arch.
> > Maybe IBM people may know what happened.
> >
> > Ananth, Jim, would you have any ideas?
>
> On powerpc, we emulate sstep whenever possible. Only recently support to
> emulate loads and stores got added. I don't have access to a powerpc box
> today... but will try to recreate the problem ASAP and see what could be
> happening in the presence of mcount.
I tried to recreate this problem on a 64-bit pSeries box without
success. Every one of the instructions in the stream at .do_fork are
emulated and work fine there -- no hangs/crashes with or without
function tracer.
Yong,
I am copying Kumar to see if he knows of any issues with 32-bit kprobes
(he wrote it) or with the function tracer, or with the toolchain itself.
You may want to check if, in the failure case, the instruction in
question is single-stepped or emulated (print out the value of
kprobe->ainsn.boostable in the post_handler) and see if you can find a
pattern to the failure.
Ananth
On Tue, 2011-06-28 at 16:11 +0530, Ananth N Mavinakayanahalli wrote:
> My access to a 32bit powerpc box is very limited. Also, embedded powerpc
> has had issues with gcc-4.6 while gcc-4.5 worked fine.
I'd work to debug this too, but I don't have access to a 32bit ppc
either. Although I've been told that people would send me one ;)
-- Steve
On Mon, Jun 27, 2011 at 6:01 PM, Ananth N Mavinakayanahalli
<[email protected]> wrote:
> On Sun, Jun 26, 2011 at 11:47:13PM +0900, Masami Hiramatsu wrote:
>> (2011/06/24 19:29), Steven Rostedt wrote:
>> > On Fri, 2011-06-24 at 17:21 +0800, Yong Zhang wrote:
>> >> Hi,
>> >>
>> >> When I use kprobe to do something, I found some wired thing.
>> >>
>> >> When CONFIG_FUNCTION_TRACER is disabled:
>> >> (gdb) disassemble do_fork
>> >> Dump of assembler code for function do_fork:
>> >> 0xc0037390 <+0>: mflr r0
>> >> 0xc0037394 <+4>: stwu r1,-64(r1)
>> >> 0xc0037398 <+8>: mfcr r12
>> >> 0xc003739c <+12>: stmw r27,44(r1)
>> >>
>> >> Then I:
>> >> modprobe kprobe_example func=do_fork offset=4
>> >> ls
>> >> Things works well.
>> >>
>> >> But when CONFIG_FUNCTION_TRACER is enabled:
>> >> (gdb) disassemble do_fork
>> >> Dump of assembler code for function do_fork:
>> >> 0xc0040334 <+0>: mflr r0
>> >> 0xc0040338 <+4>: stw r0,4(r1)
>> >> 0xc004033c <+8>: bl 0xc00109d4 <mcount>
>> >> 0xc0040340 <+12>: stwu r1,-80(r1)
>> >> 0xc0040344 <+16>: mflr r0
>> >> 0xc0040348 <+20>: stw r0,84(r1)
>> >> 0xc004034c <+24>: mfcr r12
>> >> Then I:
>> >> modprobe kprobe_example func=do_fork offset=12
>> >> ls
>> >> 'ls' will never retrun. system freeze.
>> >
>> > I'm not sure if x86 had a similar issue.
>> >
>> > Masami, have any ideas to why this happened?
>>
>> No, I don't familiar with ppc implementation. I guess
>> that single-step resume code failed to emulate the
>> instruction, but it strongly depends on ppc arch.
>> Maybe IBM people may know what happened.
>>
>> Ananth, Jim, would you have any ideas?
>
> On powerpc, we emulate sstep whenever possible. Only recently support to
> emulate loads and stores got added. I don't have access to a powerpc box
> today... but will try to recreate the problem ASAP and see what could be
> happening in the presence of mcount.
After taking more testing on it, it looks like the issue doesn't
depend on mcount
(AKA. CONFIG_FUNCTION_TRACER)
As I said in the first email, with eldk-5.0 CONFIG_FUNCTION_TRACER=n
will work well.
But when I'm using eldk-4.2[1], both will fail. But the funny thing is when I
set kprobe at several functions some works fine but some will fail. For example,
at this time do_fork() works well, but show_interrupt() will crash.
root@unknown:/root> insmod kprobe_example.ko func=show_interrupts
Planted kprobe at c009be18
root@unknown:/root> cat /proc/interrupts
pre_handler: p->addr = 0xc009be18, nip = 0xc009be18, msr = 0x29000
post_handler: p->addr = 0xc009be18, msr = 0x29000,boostable = 1
Oops: Exception in kernel mode, sig: 11 [#1]
PREEMPT MPC8536 DS
Modules linked in: kprobe_example
NIP: df159e74 LR: c0106f40 CTR: c009be18
REGS: df159d90 TRAP: 0700 Not tainted (3.0.0-rc4-00001-ge8ffcca-dirty)
MSR: 00029000 <EE,ME,CE> CR: 20202688 XER: 00000000
TASK = dfaa5340[613] 'cat' THREAD: df158000
GPR00: fffff000 df159e40 dfaa5340 df024a00 df159e78 00000000 df159f20 00000001
GPR08: c10060d0 c009be18 00029000 df159e70 00000000 1001ca74 1ffb5f00 100a01cc
GPR16: 00000000 00000000 00000000 00000000 df024a28 df159f20 00000000 dfbff080
GPR24: 10016000 00001000 df159f20 df159e78 dfbff080 df159e78 df024a00 df159e70
NIP [df159e74] 0xdf159e74
LR [c0106f40] seq_read+0x2a4/0x568
Call Trace:
[df159e40] [00029000] 0x29000 (unreliable)
[df159e74] [00000000] (null)
Instruction dump:
XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
---[ end trace 60026bfc1fe79aed ]---
Segmentation fault
Thanks,
Yong
[1]: http://ftp.denx.de/pub/eldk/4.2/
--
Only stand for myself
On Tue, Jun 28, 2011 at 6:41 PM, Ananth N Mavinakayanahalli
<[email protected]> wrote:
>
> My access to a 32bit powerpc box is very limited. Also, embedded powerpc
> has had issues with gcc-4.6 while gcc-4.5 worked fine.
I think I can do some test if you have any ideas :)
>
>> > > I'm not sure if x86 had a similar issue.
>> > >
>> > > Masami, have any ideas to why this happened?
>> >
>> > No, I don't familiar with ppc implementation. I guess
>> > that single-step resume code failed to emulate the
>> > instruction, but it strongly depends on ppc arch.
>> > Maybe IBM people may know what happened.
>> >
>> > Ananth, Jim, would you have any ideas?
>>
>> On powerpc, we emulate sstep whenever possible. Only recently support to
>> emulate loads and stores got added. I don't have access to a powerpc box
>> today... but will try to recreate the problem ASAP and see what could be
>> happening in the presence of mcount.
>
> I tried to recreate this problem on a 64-bit pSeries box without
> success. Every one of the instructions in the stream at .do_fork are
> emulated and work fine there -- no hangs/crashes with or without
> function tracer.
>
> Yong,
> I am copying Kumar to see if he knows of any issues with 32-bit kprobes
> (he wrote it) or with the function tracer, or with the toolchain itself.
>
> You may want to check if, in the failure case, the instruction in
> question is single-stepped or emulated (print out the value of
> kprobe->ainsn.boostable in the post_handler)
It's emulated:
root@unknown:/root> insmod kprobe_example.ko func=show_interrupts
Planted kprobe at c009be18
root@unknown:/root> cat /proc/interrupts
pre_handler: p->addr = 0xc009be18, nip = 0xc009be18, msr = 0x29000
post_handler: p->addr = 0xc009be18, msr = 0x29000,boostable = 1
Since commit 0016a4cf5582415849fafbf9f019dd9530824789 almost all
of the instructions are emulated.
But if we disable the emulation of stwu(so let single-stepped take it) like
below:
diff --git a/arch/powerpc/lib/sstep.c b/arch/powerpc/lib/sstep.c
index 9a52349..07f0d4a 100644
--- a/arch/powerpc/lib/sstep.c
+++ b/arch/powerpc/lib/sstep.c
@@ -1486,7 +1486,7 @@ int __kprobes emulate_step(struct pt_regs *regs,
unsigned int instr)
goto ldst_done;
case 36: /* stw */
- case 37: /* stwu */
+ //case 37: /* stwu */
val = regs->gpr[rd];
err = write_mem(val, dform_ea(instr, regs), 4, regs);
goto ldst_done;
The system will crash after single-step(looks
like the stack is currupted from the preempt_count value of
'cat/617/0x0000020a'):
pre_handler: p->addr = 0xc00ab12c, nip = 0xc00ab12c, msr = 0x29000
post_handler: p->addr = 0xc00ab12c, msr = 0x1000,boostable = -1
pre_handler: p->addr = 0xc00ab12c, nip = 0xc00ab12c, msr = 0x29000
post_handler: p->addr = 0xc00ab12c, msr = 0x1000,boostable = -1
BUG: scheduling while atomic: cat/617/0x0000020a
Modules linked in: kprobe_example [last unloaded: kprobe_example]
Call Trace:
[df157e90] [c00087c0] show_stack+0x98/0x1e4 (unreliable)
[df157ee0] [c0008938] dump_stack+0x2c/0x44
[df157ef0] [c00377c0] __schedule_bug+0x6c/0x84
[df157f00] [c060a364] schedule+0x398/0x48c
[df157f40] [c00107f4] recheck+0x0/0x24
--- Exception: c01 at 0xff1bbb8
LR = 0x1000310c
Page fault in user mode with in_atomic() = 1 mm = df01c700
NIP = ff29314 MSR = 2d000
Oops: Weird page fault, sig: 11 [#1]
PREEMPT MPC8536 DS
Modules linked in: kprobe_example [last unloaded: kprobe_example]
NIP: 0ff29314 LR: 10001944 CTR: 0ff29314
REGS: df157f50 TRAP: 0401 Tainted: G W
(3.0.0-rc4-00001-ge8ffcca-dirty)
MSR: 0002d000 <EE,PR,ME,CE> CR: 88202682 XER: 20000000
TASK = df237190[617] 'cat' THREAD: df156000
GPR00: 100018b4 bfb5c060 48007ee0 00000000 0000000e 10004354 bfb5ccde 0ff1af28
GPR08: 0202d000 48000ee8 00000000 0ff29314 1000192c
NIP [0ff29314] 0xff29314
LR [10001944] 0x10001944
Call Trace:
Kernel panic - not syncing: Fatal exception in interrupt
Call Trace:
[df157da0] [c00087c0] show_stack+0x98/0x1e4 (unreliable)
[df157df0] [c0008938] dump_stack+0x2c/0x44
[df157e00] [c0042a80] panic+0xc4/0x1f4
[df157e60] [c000c4e0] die+0x1fc/0x22c
[df157e90] [c060e4a4] do_page_fault+0x130/0x4c4
[df157f40] [c00100fc] handle_page_fault+0xc/0x80
--- Exception: 401 at 0xff29314
LR = 0x10001944
Thanks,
Yong
--
Only stand for myself
On Wed, Jun 29, 2011 at 02:23:28PM +0800, Yong Zhang wrote:
> On Mon, Jun 27, 2011 at 6:01 PM, Ananth N Mavinakayanahalli
> <[email protected]> wrote:
> > On Sun, Jun 26, 2011 at 11:47:13PM +0900, Masami Hiramatsu wrote:
> >> (2011/06/24 19:29), Steven Rostedt wrote:
> >> > On Fri, 2011-06-24 at 17:21 +0800, Yong Zhang wrote:
> >> >> Hi,
> >> >>
> >> >> When I use kprobe to do something, I found some wired thing.
> >> >>
> >> >> When CONFIG_FUNCTION_TRACER is disabled:
> >> >> (gdb) disassemble do_fork
> >> >> Dump of assembler code for function do_fork:
> >> >> ? ?0xc0037390 <+0>: ? ? ? ?mflr ? ?r0
> >> >> ? ?0xc0037394 <+4>: ? ? ? ?stwu ? ?r1,-64(r1)
> >> >> ? ?0xc0037398 <+8>: ? ? ? ?mfcr ? ?r12
> >> >> ? ?0xc003739c <+12>: ? ? ? stmw ? ?r27,44(r1)
> >> >>
> >> >> Then I:
> >> >> modprobe kprobe_example func=do_fork offset=4
> >> >> ls
> >> >> Things works well.
> >> >>
> >> >> But when CONFIG_FUNCTION_TRACER is enabled:
> >> >> (gdb) disassemble do_fork
> >> >> Dump of assembler code for function do_fork:
> >> >> ? ?0xc0040334 <+0>: ? ? ? ?mflr ? ?r0
> >> >> ? ?0xc0040338 <+4>: ? ? ? ?stw ? ? r0,4(r1)
> >> >> ? ?0xc004033c <+8>: ? ? ? ?bl ? ? ?0xc00109d4 <mcount>
> >> >> ? ?0xc0040340 <+12>: ? ? ? stwu ? ?r1,-80(r1)
> >> >> ? ?0xc0040344 <+16>: ? ? ? mflr ? ?r0
> >> >> ? ?0xc0040348 <+20>: ? ? ? stw ? ? r0,84(r1)
> >> >> ? ?0xc004034c <+24>: ? ? ? mfcr ? ?r12
> >> >> Then I:
> >> >> modprobe kprobe_example func=do_fork offset=12
> >> >> ls
> >> >> 'ls' will never retrun. system freeze.
> >> >
> >> > I'm not sure if x86 had a similar issue.
> >> >
> >> > Masami, have any ideas to why this happened?
> >>
> >> No, I don't familiar with ppc implementation. I guess
> >> that single-step resume code failed to emulate the
> >> instruction, but it strongly depends on ppc arch.
> >> Maybe IBM people may know what happened.
> >>
> >> Ananth, Jim, would you have any ideas?
> >
> > On powerpc, we emulate sstep whenever possible. Only recently support to
> > emulate loads and stores got added. I don't have access to a powerpc box
> > today... but will try to recreate the problem ASAP and see what could be
> > happening in the presence of mcount.
>
> After taking more testing on it, it looks like the issue doesn't
> depend on mcount
> (AKA. CONFIG_FUNCTION_TRACER)
>
> As I said in the first email, with eldk-5.0 CONFIG_FUNCTION_TRACER=n
> will work well.
>
> But when I'm using eldk-4.2[1], both will fail. But the funny thing is when I
> set kprobe at several functions some works fine but some will fail. For example,
> at this time do_fork() works well, but show_interrupt() will crash.
Certain functions are off limits for probing -- look for __kprobe
annotations in the kernel. Some such functions are arch specific, but
show_interrupts() would definitely not be one of them. It works fine on
my (64bit) test box.
At this time, I think your best bet is to work with the eldk folks to
narrow down the problem. Given the current set of data, I am inclined to
think it could be an eldk bug, not a kernel one.
Ananth
On Wed, Jun 29, 2011 at 2:46 PM, Ananth N Mavinakayanahalli
<[email protected]> wrote:
>
> Certain functions are off limits for probing -- look for __kprobe
Yup.
> annotations in the kernel. Some such functions are arch specific, but
> show_interrupts() would definitely not be one of them. It works fine on
> my (64bit) test box.
>
> At this time, I think your best bet is to work with the eldk folks to
> narrow down the problem.
I'll give a try :)
> Given the current set of data, I am inclined to
> think it could be an eldk bug, not a kernel one.
Maybe, but the fact is if I don't use kprobe, things works
very well.
I'll be back if there is any update :)
Thanks,
Yong
--
Only stand for myself