While testing, I noticed that Sysrq p is silently ignored on current RT
kernels with RT preemption: The syslog contains a message that Sysrq p
was pressed, but no registers are printed.
A vanilla 2.6.11 with the same config (except for the RT-specific
settings) on the same hardware (i386) gives correct Sysrq p output.
The keyboard is connected via USB.
I checked the code: The sysrq p function is passed a "struct pt_regs *"
arg, and only produces output if this pointer is non-NULL. For RT
kernels, it is always NULL (I added a printk).
This "struct pt_regs *" travels all the way from the actual keyboard
interrupt handler (it is passed as an arg to "usb_kbd_irq"), but I was
not able to trace it back any further in the source.
Is this a known problem?
Is there any workaround? (Sysrq p would be very helpful for debugging!)
Thanks in advance for any hints!
--
Klaus Kusche (Software Development - Control Systems)
KEBA AG Gewerbepark Urfahr, A-4041 Linz, Austria (Europe)
Tel: +43 / 732 / 7090-3120 Fax: +43 / 732 / 7090-6301
E-Mail: [email protected] WWW: http://www.keba.com
* kus Kusche Klaus <[email protected]> wrote:
> While testing, I noticed that Sysrq p is silently ignored on current
> RT kernels with RT preemption: The syslog contains a message that
> Sysrq p was pressed, but no registers are printed.
yes, that's because the keyboard interrupt is 'threaded' - hence there's
no 'interrupted stack' to print a backtrace of. You should be able to
see all (including currently running) task's backtraces in SysRq-T
output.
are you trying to use it to debug a particular bug?
Ingo
> * kus Kusche Klaus <[email protected]> wrote:
>
> > While testing, I noticed that Sysrq p is silently ignored
> on current
> > RT kernels with RT preemption: The syslog contains a message that
> > Sysrq p was pressed, but no registers are printed.
>
> yes, that's because the keyboard interrupt is 'threaded' -
> hence there's
> no 'interrupted stack' to print a backtrace of. You should be able to
> see all (including currently running) task's backtraces in SysRq-T
> output.
>
> are you trying to use it to debug a particular bug?
No, not yet.
I've been asked to analyze the various tools and possibilities available
to debug an RT kernel.
Up to now, what I've found is not too impressive:
* Plain GDB can be used to display the current value of kernel variables
symbolically, but no more: It won't tell anything about the kernel's
current activity.
* kgdb and kdb seem to be deeply incompatible with the RT patches (they
mess with the scheduler, interrupts etc.), applying their patches to an
RT tree fails quite impressively.
* LKCD is too heavy for debugging embedded systems (which do not have
swap partitions or other permanent memory to spare); for netdump, I was
unable to find any recent and working download (I found the server, but
where is the client-side kernel code?); and kdump-via-kexec is "not yet
there". Moreover, most of them are for i386 and won't support ARM (or
PPC).
So Sysrq was one of my hopes to give developers an easy tool to check
quickly where their kernel is, if it still moves or loops, what the
reg's are, ...
Sysrq t works, but produces by far too much output for that purpose -
it's hard to get any information (especially if things are still
changing or already frozen) from Sysrq t "at a quick glance".
Any other hints for debugging?
--
Klaus Kusche (Software Development - Control Systems)
KEBA AG Gewerbepark Urfahr, A-4041 Linz, Austria (Europe)
Tel: +43 / 732 / 7090-3120 Fax: +43 / 732 / 7090-6301
E-Mail: [email protected] WWW: http://www.keba.com
* kus Kusche Klaus <[email protected]> wrote:
> I've been asked to analyze the various tools and possibilities
> available to debug an RT kernel.
>
> Up to now, what I've found is not too impressive:
> * Plain GDB can be used to display the current value of kernel variables
> symbolically, but no more: It won't tell anything about the kernel's
> current activity.
> * kgdb and kdb seem to be deeply incompatible with the RT patches (they
> mess with the scheduler, interrupts etc.), applying their patches to an
> RT tree fails quite impressively.
kgdb is in the -mm tree, and there are periodic ports to the -mm tree.
Someone used it too on PREEMPT_RT - with some success. There's no deep
incompatibility with the -RT kernel - just line-for-line collisions.
Ingo
On Mon, 2005-05-09 at 16:02 +0200, Ingo Molnar wrote:
> * kus Kusche Klaus <[email protected]> wrote:
>
> > I've been asked to analyze the various tools and possibilities
> > available to debug an RT kernel.
> >
> > Up to now, what I've found is not too impressive:
> > * Plain GDB can be used to display the current value of kernel variables
> > symbolically, but no more: It won't tell anything about the kernel's
> > current activity.
> > * kgdb and kdb seem to be deeply incompatible with the RT patches (they
> > mess with the scheduler, interrupts etc.), applying their patches to an
> > RT tree fails quite impressively.
>
> kgdb is in the -mm tree, and there are periodic ports to the -mm tree.
> Someone used it too on PREEMPT_RT - with some success. There's no deep
> incompatibility with the -RT kernel - just line-for-line collisions.
That was me. And it did work. Ingo is right, I only had to make some
trivial changes to apply the patch.
Lee