2005-11-22 13:07:57

by folkert

[permalink] [raw]
Subject: capturing oopses

Hi,

My 2.6.14 system occasionally crashes; gives a kernel panic. Of course I
would like to report it. Now the system locks up hard so I can't copy
the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
don't work for 2.6.14 it seems. So: what should I do? Get my digicam and
take a picture of the display?


Folkert van Heusden

--
Try MultiTail! Multiple windows with logfiles, filtered with regular
expressions, colored output, etc. etc. http://www.vanheusden.com/multitail/
----------------------------------------------------------------------
Get your PGP/GPG key signed at http://www.biglumber.com!
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com


2005-11-22 14:08:37

by bert hubert

[permalink] [raw]
Subject: Re: capturing oopses

On Tue, Nov 22, 2005 at 02:07:54PM +0100, Folkert van Heusden wrote:
> My 2.6.14 system occasionally crashes; gives a kernel panic. Of course I
> would like to report it. Now the system locks up hard so I can't copy
> the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
> don't work for 2.6.14 it seems. So: what should I do? Get my digicam and
> take a picture of the display?

Try the serial port- you can get a copy of the console on ttyS0.

--
http://www.PowerDNS.com Open source, database driven DNS Software
http://netherlabs.nl Open and Closed source services

2005-11-23 01:28:09

by Brice Goglin

[permalink] [raw]
Subject: Re: capturing oopses

bert hubert wrote:

>On Tue, Nov 22, 2005 at 02:07:54PM +0100, Folkert van Heusden wrote:
>
>
>>My 2.6.14 system occasionally crashes; gives a kernel panic. Of course I
>>would like to report it. Now the system locks up hard so I can't copy
>>the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
>>don't work for 2.6.14 it seems. So: what should I do? Get my digicam and
>>take a picture of the display?
>>
>>
>
>Try the serial port- you can get a copy of the console on ttyS0.
>
>
>
netconsole works great too.

Brice

2005-11-23 10:55:27

by folkert

[permalink] [raw]
Subject: Re: capturing oopses

> > My 2.6.14 system occasionally crashes; gives a kernel panic. Of course I
> > would like to report it. Now the system locks up hard so I can't copy
> > the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
> > don't work for 2.6.14 it seems. So: what should I do? Get my digicam and
> > take a picture of the display?
> Try the serial port- you can get a copy of the console on ttyS0.

No serial ports free unfortunately.


Folkert van Heusden

--
Try MultiTail! Multiple windows with logfiles, filtered with regular
expressions, colored output, etc. etc. http://www.vanheusden.com/multitail/
----------------------------------------------------------------------
Get your PGP/GPG key signed at http://www.biglumber.com!
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com

2005-11-23 11:17:44

by bert hubert

[permalink] [raw]
Subject: Re: capturing oopses

On Wed, Nov 23, 2005 at 11:55:24AM +0100, Folkert van Heusden wrote:
> > Try the serial port- you can get a copy of the console on ttyS0.
>
> No serial ports free unfortunately.

Then I suggest you whip out your trusty pencil.

--
http://www.PowerDNS.com Open source, database driven DNS Software
http://netherlabs.nl Open and Closed source services

2005-11-26 15:56:59

by Adrian Bunk

[permalink] [raw]
Subject: Re: capturing oopses

On Tue, Nov 22, 2005 at 02:07:54PM +0100, Folkert van Heusden wrote:

> Hi,

Hi Folkert,

> My 2.6.14 system occasionally crashes; gives a kernel panic. Of course I
> would like to report it. Now the system locks up hard so I can't copy
> the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
> don't work for 2.6.14 it seems. So: what should I do? Get my digicam and
> take a picture of the display?

yes, digicams have become a common tool for reporting Oops'es.

> Folkert van Heusden

cu
Adrian

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

2005-11-26 19:34:05

by Ville Herva

[permalink] [raw]
Subject: Re: capturing oopses

On Sat, Nov 26, 2005 at 04:56:56PM +0100, you [Adrian Bunk] wrote:
> On Tue, Nov 22, 2005 at 02:07:54PM +0100, Folkert van Heusden wrote:
>
> > Hi,
>
> Hi Folkert,
>
> > My 2.6.14 system occasionally crashes; gives a kernel panic. Of course I
> > would like to report it. Now the system locks up hard so I can't copy
> > the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
> > don't work for 2.6.14 it seems. So: what should I do? Get my digicam and
> > take a picture of the display?
>
> yes, digicams have become a common tool for reporting Oops'es.

Speaking of which, does anybody know a feasible (as in "not too much harder
than manually typing it in manually") way to OCR characters from vga text
mode screen captures - or even digican shots?

The vga text mode captures are from a remote administration interface (such
as HP RILOE or vmware gsx console) so they are pixel perfect and OCR should
be doable. The digican shots on the other hand... Well at least it would
have hack value :).

(My personal opinion is that Linus' unwillingness to include anything like
kmsgdump (http://www.xenotime.net/linux/kmsgdump/) is somewhat unfortunate.)


-- v --

[email protected]

2005-11-27 01:23:49

by Bodo Eggert

[permalink] [raw]
Subject: Re: capturing oopses

Ville Herva <[email protected]> wrote:

> Speaking of which, does anybody know a feasible (as in "not too much harder
> than manually typing it in manually") way to OCR characters from vga text
> mode screen captures - or even digican shots?
>
> The vga text mode captures are from a remote administration interface (such
> as HP RILOE or vmware gsx console) so they are pixel perfect and OCR should
> be doable.

If it's a 640 pixel width image and you've got the right font loaded, the
VGA function 0x08 will get you the character at the current cursor position.
It's also used by PRTSCR, so pressing it in a DOS viewer may dump the text
to LPT1. If you redirected it to a serial console, you might also catch it.
(I never tried).
--
Ich danke GMX daf?r, die Verwendung meiner Adressen mittels per SPF
verbreiteten L?gen zu sabotieren.

2005-11-27 12:16:09

by Ville Herva

[permalink] [raw]
Subject: Re: capturing oopses

On Sun, Nov 27, 2005 at 02:10:59AM +0100, you [Bodo Eggert] wrote:
>
> If it's a 640 pixel width image and you've got the right font loaded, the
> VGA function 0x08 will get you the character at the current cursor position.
> It's also used by PRTSCR, so pressing it in a DOS viewer may dump the text
> to LPT1. If you redirected it to a serial console, you might also catch it.
> (I never tried).

I meant that I have the image as a PNG (whatever), captured from the display
of the remote control program (HP RILOE, VMWare GSX Console, VNC, whatever).
I'm aware that the VGA display memory holds the characters (in addition to
pixels) in text mode, but the actual machine may be dead at this point - OS
crash, hw failure, whatever. The oops output is on the screen - just not in
palatible text format.


-- v --

[email protected]

2005-11-27 13:55:30

by Sander

[permalink] [raw]
Subject: Re: capturing oopses

Ville Herva wrote (ao):
> I meant that I have the image as a PNG (whatever), captured from the display
> of the remote control program (HP RILOE, VMWare GSX Console, VNC, whatever).
> I'm aware that the VGA display memory holds the characters (in addition to
> pixels) in text mode, but the actual machine may be dead at this point - OS
> crash, hw failure, whatever. The oops output is on the screen - just not in
> palatible text format.

Maybe 'gorc' can help you. I don't have an url, but it is package 'gorc'
in Debian.

I had little success with a digicam photo of lightgray text on
not-quite-white paper, even after trying to improve things with gimp,
but it might work if you have the oops on screen.

Then again, if you have it on screen, you might as well put it on the
web.

--
Humilis IT Services and Solutions
http://www.humilis.net

2005-11-27 14:45:04

by Ville Herva

[permalink] [raw]
Subject: Re: capturing oopses

On Sun, Nov 27, 2005 at 02:55:41PM +0100, you [Sander] wrote:
>
> Maybe 'gorc' can help you. I don't have an url, but it is package 'gorc'
> in Debian.
>
> I had little success with a digicam photo of lightgray text on
> not-quite-white paper, even after trying to improve things with gimp,
> but it might work if you have the oops on screen.

Seems promising, but the results are not that great. See

http://v.iki.fi/~vherva/tmp/shot.png
and
http://v.iki.fi/~vherva/tmp/shot.txt

the gray level setting didn't help much.

> Then again, if you have it on screen, you might as well put it on the
> web.

Sometimes you need to ksymoops the dump (for older kernels) or paste some
parts to some program.

In theory, converting a vga text screen capture shouldn't be hard, but in
practise... Well, it's much more beneficial to me to learn to type
passably...


-- v --

[email protected]

2005-11-27 14:56:57

by Ville Herva

[permalink] [raw]
Subject: Re: capturing oopses

On Sun, Nov 27, 2005 at 04:44:57PM +0200, you [Ville Herva] wrote:
> On Sun, Nov 27, 2005 at 02:55:41PM +0100, you [Sander] wrote:
> >
> > Maybe 'gorc' can help you. I don't have an url, but it is package 'gorc'
> > in Debian.
> >
> > I had little success with a digicam photo of lightgray text on
> > not-quite-white paper, even after trying to improve things with gimp,
> > but it might work if you have the oops on screen.
>
> Seems promising, but the results are not that great. See
>
> http://v.iki.fi/~vherva/tmp/shot.png
> and
> http://v.iki.fi/~vherva/tmp/shot.txt
>
> the gray level setting didn't help much.

Ok, if I invert the image (black on white), I get
http://v.iki.fi/~vherva/tmp/shot2.txt
which is fairly usable.


-- v --

[email protected]

2005-11-28 04:41:05

by Randy Dunlap

[permalink] [raw]
Subject: Re: capturing oopses

On Sat, 26 Nov 2005 21:33:58 +0200 Ville Herva wrote:

> On Sat, Nov 26, 2005 at 04:56:56PM +0100, you [Adrian Bunk] wrote:
> > On Tue, Nov 22, 2005 at 02:07:54PM +0100, Folkert van Heusden wrote:
> >
> > > Hi,
> >
> > Hi Folkert,
> >
> > > My 2.6.14 system occasionally crashes; gives a kernel panic. Of course I
> > > would like to report it. Now the system locks up hard so I can't copy
> > > the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
> > > don't work for 2.6.14 it seems. So: what should I do? Get my digicam and
> > > take a picture of the display?
> >
> > yes, digicams have become a common tool for reporting Oops'es.
>
> Speaking of which, does anybody know a feasible (as in "not too much harder
> than manually typing it in manually") way to OCR characters from vga text
> mode screen captures - or even digican shots?
>
> The vga text mode captures are from a remote administration interface (such
> as HP RILOE or vmware gsx console) so they are pixel perfect and OCR should
> be doable. The digican shots on the other hand... Well at least it would
> have hack value :).
>
> (My personal opinion is that Linus' unwillingness to include anything like
> kmsgdump (http://www.xenotime.net/linux/kmsgdump/) is somewhat unfortunate.)

BTW, status of that: it needs a little work to be more reliable.
(It hangs sometimes when switching from protected to real mode.)
I'm hoping that some of the APIC/IOAPIC/PIC patches that are being
done for kdump will also help kmsgdump. I'll be working more on it
in the next few weeks/months.

so yes, when it's working, it's very useful IMO.

---
~Randy

2005-11-28 05:14:07

by Maneesh Soni

[permalink] [raw]
Subject: Re: capturing oopses

On Tue, Nov 22, 2005 at 02:07:54PM +0100, Folkert van Heusden wrote:

> the stacktrace. The crash dump patches mentioned in oops-tracing.txt all
> don't work for 2.6.14 it seems. So: what should I do? Get my digicam and

could you try kdump (Documentation/kdump/kdump.txt) to capture the crash
dump. It is part of mainline kernel atleast for i386 architecture. It
should work for 2.6.14 also.

Thanks
Maneesh

--
Maneesh Soni
Linux Technology Center,
IBM India Software Labs,
Bangalore, India
email: [email protected]
Phone: 91-80-25044990

2005-11-28 16:21:14

by mark gross

[permalink] [raw]
Subject: Re: capturing oopses

On Sunday 27 November 2005 20:41, Randy.Dunlap wrote:
> On Sat, 26 Nov 2005 21:33:58 +0200 Ville Herva wrote:
> > On Sat, Nov 26, 2005 at 04:56:56PM +0100, you [Adrian Bunk] wrote:
> > > On Tue, Nov 22, 2005 at 02:07:54PM +0100, Folkert van Heusden wrote:
> > > > Hi,
> > >
> > > Hi Folkert,
> > >
> > > > My 2.6.14 system occasionally crashes; gives a kernel panic. Of
> > > > course I would like to report it. Now the system locks up hard so I
> > > > can't copy the stacktrace. The crash dump patches mentioned in
> > > > oops-tracing.txt all don't work for 2.6.14 it seems. So: what should
> > > > I do? Get my digicam and take a picture of the display?
> > >
> > > yes, digicams have become a common tool for reporting Oops'es.
> >
> > Speaking of which, does anybody know a feasible (as in "not too much
> > harder than manually typing it in manually") way to OCR characters from
> > vga text mode screen captures - or even digican shots?
> >
> > The vga text mode captures are from a remote administration interface
> > (such as HP RILOE or vmware gsx console) so they are pixel perfect and
> > OCR should be doable. The digican shots on the other hand... Well at
> > least it would have hack value :).
> >
> > (My personal opinion is that Linus' unwillingness to include anything
> > like kmsgdump (http://www.xenotime.net/linux/kmsgdump/) is somewhat
> > unfortunate.)
>
> BTW, status of that: it needs a little work to be more reliable.
> (It hangs sometimes when switching from protected to real mode.)
> I'm hoping that some of the APIC/IOAPIC/PIC patches that are being
> done for kdump will also help kmsgdump. I'll be working more on it
> in the next few weeks/months.
>
> so yes, when it's working, it's very useful IMO.
>

You know some platforms that perserve the memory above some addresses across
warm boots. For such platforms, one could reserve a buffer in that area can
copy the sys log buffer to it on panic along with a bit pattern that could be
searched for upon the next boot.

Additionally some platforms have flash parts that could be used as a
persistant store for such data in the same manner. I've done this on an
XScale platform using an ancient (2.4.10) kernel. I've always thought it
would be a handy thing for the newer kernels for PC's, if we could come up
with a semi-platform independent way of identifying a few pages that would
survive a warm boot.

--mgross

2005-11-28 16:42:31

by Chris Friesen

[permalink] [raw]
Subject: Re: capturing oopses

mgross wrote:

> You know some platforms that perserve the memory above some addresses across
> warm boots. For such platforms, one could reserve a buffer in that area can
> copy the sys log buffer to it on panic along with a bit pattern that could be
> searched for upon the next boot.

I've worked on a few blades that had this. Serial console wasn't going
to be available in the field, so being able to just dump debug
information to persistent memory was really nice.

Chris

2005-11-28 19:35:57

by Ville Herva

[permalink] [raw]
Subject: Re: capturing oopses

On Mon, Nov 28, 2005 at 08:20:02AM -0800, you [mgross] wrote:
>
> You know some platforms that perserve the memory above some addresses across
> warm boots. For such platforms, one could reserve a buffer in that area can
> copy the sys log buffer to it on panic along with a bit pattern that could be
> searched for upon the next boot.

I think there was a patch to do this a couple of years ago.

The only references I can find now are:

http://groups.google.com/groups?q=%22Utility+module+to+capture+OOPS+output+over+reboot%22&hl=en&selm=fa.fbd0l7v.14hau3n%40ifi.uio.no&rnum=1)
http://marc.theaimsgroup.com/?l=linux-kernel&m=93077831203565&w=2


-- v --

[email protected]