2010-07-08 11:51:59

by Robert P. J. Day

[permalink] [raw]
Subject: why can't i use gdb to dump module variables anymore?


trying very hard to make a long story short, documenting how to use
gdb to poke around in a loaded module. the important part of the
module (to demonstrate the visibility of different types of
variables):

static int rpjday_1;
int rpjday_2 = 20;
int rpjday_3 = 30;

EXPORT_SYMBOL(rpjday_3);


once i load that module, i can see the good stuff in /proc/kallsyms:

$ grep rpjday /proc/kallsyms
ffffffffa007c090 r __ksymtab_rpjday_3 [crash_gdb]
ffffffffa007c0a8 r __kstrtab_rpjday_3 [crash_gdb]
ffffffffa007c0a0 r __kcrctab_rpjday_3 [crash_gdb]
ffffffffa007c0b4 d rpjday_2 [crash_gdb]
ffffffffa007c0b8 D rpjday_3 [crash_gdb]
$

i can examine the salient files under
/sys/module/crash_gdb/sections/, like .data

0xffffffffa007c0b4

so here's a couple questions.

if i want to add that module's symbol table info to my running gdb
session, i would use "add-symbol-file" but the first arg *must* be the
address of the text section and this module has *no* .text section.
so what do i use for that value? the address of the .exit.text
section?

and if i try to load:

(gdb) add-symbol-file /tmp/crash_gdb.ko 0xffffffffa007c000 \
-s .data 0xffffffffa007c0b4
add symbol table from file "/tmp/crash_gdb.ko" at
.text_addr = 0xffffffffa007c000
.data_addr = 0xffffffffa007c0b4
(y or n) y
Reading symbols from /tmp/crash_gdb.ko...done.
(gdb)

it certainly *looks* like that worked, and:

(gdb) whatis rpjday_2
type = int
(gdb) whatis rpjday_3
type = int

looks good, but here's where things go wrong:

(gdb) p rpjday_2
Cannot access memory at address 0xffffffffa007c0b4
(gdb) p rpjday_3
Cannot access memory at address 0xffffffffa007c0b8

that sort of thing used to work when i was testing this on a 32-bit
fedora system, but i'm now on 64-but ubuntu and i don't see why those
last couple commands no longer work. the addresses certainly match
what's in /proc/kallsyms, and i don't recall having to do anything
special last time. have i omitted a critical compile-time option for
this? thanks.

rday

--

========================================================================
Robert P. J. Day Waterloo, Ontario, CANADA

Top-notch, inexpensive online Linux/OSS/kernel courses
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================


2010-08-09 16:06:51

by Robert P. J. Day

[permalink] [raw]
Subject: Re: why can't i use gdb to dump module variables anymore?

On Sun, 8 Aug 2010, Jon Masters wrote:

>
> On Jul 8, 2010, at 7:51 AM, Robert P. J. Day wrote:
> >
> > (gdb) p rpjday_2
> > Cannot access memory at address 0xffffffffa007c0b4
> > (gdb) p rpjday_3
> > Cannot access memory at address 0xffffffffa007c0b8
>
> Did you ever get a reply to this one? Were you trying to do this
> locally, by loading a kernel core or using kgdb?

never figured out what was happening here, used to work just fine
for simple gdb debugging of the kernel; nowadays (at least on ubuntu),
not so much. anyway, i'm in bahstahn for linuxcon, staying at the
onyx hotel for first 3 days. my girlfriend is doing a bofs session:

http://events.linuxfoundation.org/linuxcon2010/mckinnon

anyway, regarding gdb, this was based on a column i wrote last
year:

http://www.linux.com/learn/linux-training/33991-the-kernel-newbie-corner-kernel-and-module-debugging-with-gdb

where it worked just fine -- it's the technique described in LDD3.
but now, on ubuntu 10.04, weirdness. i haven't looked at it closely
again for a while, but if you have any ideas, i'm open to suggestions.
maybe some odd security/selinux/??? setting?

rday


--

========================================================================
Robert P. J. Day Waterloo, Ontario, CANADA

Top-notch, inexpensive online Linux/OSS/kernel courses
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================

2010-08-10 12:01:41

by Jason Wessel

[permalink] [raw]
Subject: Re: why can't i use gdb to dump module variables anymore?

On 08/09/2010 11:06 AM, Robert P. J. Day wrote:
> On Sun, 8 Aug 2010, Jon Masters wrote:
>
>
>> On Jul 8, 2010, at 7:51 AM, Robert P. J. Day wrote:
>>
>>> (gdb) p rpjday_2
>>> Cannot access memory at address 0xffffffffa007c0b4
>>> (gdb) p rpjday_3
>>> Cannot access memory at address 0xffffffffa007c0b8
>>>
>> Did you ever get a reply to this one? Were you trying to do this
>> locally, by loading a kernel core or using kgdb?
>>
>
> never figured out what was happening here, used to work just fine
> for simple gdb debugging of the kernel; nowadays (at least on ubuntu),
> not so much. anyway, i'm in bahstahn for linuxcon, staying at the
> onyx hotel for first 3 days. my girlfriend is doing a bofs session:
>
> http://events.linuxfoundation.org/linuxcon2010/mckinnon
>
> anyway, regarding gdb, this was based on a column i wrote last
> year:
>
> http://www.linux.com/learn/linux-training/33991-the-kernel-newbie-corner-kernel-and-module-debugging-with-gdb
>
> where it worked just fine -- it's the technique described in LDD3.
> but now, on ubuntu 10.04, weirdness. i haven't looked at it closely
> again for a while, but if you have any ideas, i'm open to suggestions.
> maybe some odd security/selinux/??? setting?
>

I took a brief look at the instructions you had a the URL. It seemed
like something that should probably work ok unless that memory address
is really not accessible via /proc/kcore.

Depending on the age of the ubuntu system and if you have a serial port
or not, kgdb is actually enabled in the kernel and you could certainly
try the same test there.

It should also be possible to try out a gdb with kernel module awareness
on the /proc/kcore. If you want to try and debug it at some point
perhaps we'll cross paths at LinuxCON.

Jason

2010-08-10 12:12:25

by Robert P. J. Day

[permalink] [raw]
Subject: Re: why can't i use gdb to dump module variables anymore?

On Tue, 10 Aug 2010, Jason Wessel wrote:

> On 08/09/2010 11:06 AM, Robert P. J. Day wrote:
> > On Sun, 8 Aug 2010, Jon Masters wrote:
> >
> >
> >> On Jul 8, 2010, at 7:51 AM, Robert P. J. Day wrote:
> >>
> >>> (gdb) p rpjday_2
> >>> Cannot access memory at address 0xffffffffa007c0b4
> >>> (gdb) p rpjday_3
> >>> Cannot access memory at address 0xffffffffa007c0b8
> >>>
> >> Did you ever get a reply to this one? Were you trying to do this
> >> locally, by loading a kernel core or using kgdb?
> >>
> >
> > never figured out what was happening here, used to work just
> > fine for simple gdb debugging of the kernel; nowadays (at least on
> > ubuntu), not so much. anyway, i'm in bahstahn for linuxcon,
> > staying at the onyx hotel for first 3 days. my girlfriend is
> > doing a bofs session:
> >
> > http://events.linuxfoundation.org/linuxcon2010/mckinnon
> >
> > anyway, regarding gdb, this was based on a column i wrote last
> > year:
> >
> > http://www.linux.com/learn/linux-training/33991-the-kernel-newbie-corner-kernel-and-module-debugging-with-gdb
> >
> > where it worked just fine -- it's the technique described in LDD3.
> > but now, on ubuntu 10.04, weirdness. i haven't looked at it
> > closely again for a while, but if you have any ideas, i'm open to
> > suggestions. maybe some odd security/selinux/??? setting?
> >
>
> I took a brief look at the instructions you had a the URL. It
> seemed like something that should probably work ok unless that
> memory address is really not accessible via /proc/kcore.

i've used that technique before as a simple way to examine kernel
data on a running system, and it always worked fine before.

> Depending on the age of the ubuntu system and if you have a serial
> port or not, kgdb is actually enabled in the kernel and you could
> certainly try the same test there.

i can give that a shot later, certainly.

> It should also be possible to try out a gdb with kernel module
> awareness on the /proc/kcore. If you want to try and debug it at
> some point perhaps we'll cross paths at LinuxCON.

i haven't looked at this issue for a while, but i'd certainly like
to figure out why what used to work before has stopped working. this
is something i like to show my beginner kernel programming students,
and it's moderately frustrating that i can't get it to work anymore.

i'm sure it's something idiotic i'm doing. and i'm around linuxcon
all week, so yes, let's bump into each other.

rday


--

========================================================================
Robert P. J. Day Waterloo, Ontario, CANADA

Top-notch, inexpensive online Linux/OSS/kernel courses
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================