--- ./kernel/irq/spurious.c.irqlockup 2006-11-09 11:19:10.000000000 +0300
+++ ./kernel/irq/spurious.c 2006-11-10 16:53:38.000000000 +0300
@@ -147,7 +147,11 @@ void note_interrupt(unsigned int irq, st
if (unlikely(irqfixup)) {
/* Don't punish working computers */
if ((irqfixup == 2 && irq == 0) || action_ret == IRQ_NONE) {
- int ok = misrouted_irq(irq);
+ int ok;
+
+ spin_unlock(&desc->lock);
+ ok = misrouted_irq(irq);
+ spin_lock(&desc->lock);
if (action_ret == IRQ_NONE)
desc->irqs_unhandled -= ok;
}
On Fri, 2006-11-10 at 16:55 +0300, Pavel Emelianov wrote:
> - int ok = misrouted_irq(irq);
> + int ok;
> +
> + spin_unlock(&desc->lock);
> + ok = misrouted_irq(irq);
> + spin_lock(&desc->lock);
your fix looks reasonable to me - it's a thinko to call misrouted_irq()
with the descriptor lock still held. (btw., how did you find it -
lockdep spinlock debugging or NMI watchdog?)
Acked-by: Ingo Molnar <[email protected]>
Ingo
Ingo Molnar wrote:
> On Fri, 2006-11-10 at 16:55 +0300, Pavel Emelianov wrote:
>> - int ok = misrouted_irq(irq);
>> + int ok;
>> +
>> + spin_unlock(&desc->lock);
>> + ok = misrouted_irq(irq);
>> + spin_lock(&desc->lock);
>
> your fix looks reasonable to me - it's a thinko to call misrouted_irq()
> with the descriptor lock still held. (btw., how did you find it -
> lockdep spinlock debugging or NMI watchdog?)
It was NMI watchdog who reported the deadlock. With lockdep
turned on it wouldn't be caught - local_irq_enable_in_hardirq()
is nothing but a "do { } while (0)" if CONFIG_LOCKDEP=y :)
The second issue (with 2 cpus involved) was discovered
by code examining.
> Acked-by: Ingo Molnar <[email protected]>
>
> Ingo
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
On Fri, Nov 10, 2006 at 04:55:48PM +0300, Pavel Emelianov wrote:
> As the second lock on booth CPUs is taken before checking that
> this irq is being handled in another processor this may cause
> a deadlock. This issue is only theoretical.
>
> I propose the attached patch to fix booth problems: when trying
> to handle misrouted IRQ active desc->lock may be unlocked.
>
> Please comment.
> --- ./kernel/irq/spurious.c.irqlockup 2006-11-09 11:19:10.000000000 +0300
> +++ ./kernel/irq/spurious.c 2006-11-10 16:53:38.000000000 +0300
> @@ -147,7 +147,11 @@ void note_interrupt(unsigned int irq, st
> if (unlikely(irqfixup)) {
> /* Don't punish working computers */
> if ((irqfixup == 2 && irq == 0) || action_ret == IRQ_NONE) {
> - int ok = misrouted_irq(irq);
> + int ok;
> +
> + spin_unlock(&desc->lock);
> + ok = misrouted_irq(irq);
> + spin_lock(&desc->lock);
Hi Pavel,
Till -rc5, I was able to boot a kernel with irqpoll option and in -rc6 I
can't. It simply hangs. I suspect it is realted to this change. I have yet
to confirm that. But before that one observation.
Not at every place note_interrupt() is called with desc->lock() held. For
example, handle_level_irq(). I enabled spin lock debugging and I run into
following BUG().
PID hash table entries: 256 (order: 8, 2048 bytes)
time.c: Using 3.579545 MHz WALL PM GTOD PIT/TSC timer.
time.c: Detected 3000.218 MHz processor.
=====================================
[ BUG: bad unlock balance detected! ]
-------------------------------------
swapper/0 is trying to release lock (&irq_desc_lock_class) at:
[<ffffffff8104c673>] note_interrupt+0x7a/0x22b
but there are no more locks to release!
other info that might help us debug this:
no locks held by swapper/0.
stack backtrace:
Call Trace:
[<ffffffff8100a6f9>] show_trace+0x34/0x47
[<ffffffff8100a71e>] dump_stack+0x12/0x17
[<ffffffff8103caba>] print_unlock_inbalance_bug+0xfb/0x106
[<ffffffff8103e6e5>] lock_release+0x89/0x128
[<ffffffff81332d96>] _spin_unlock+0x17/0x20
[<ffffffff8104c673>] note_interrupt+0x7a/0x22b
[<ffffffff8104d131>] handle_level_irq+0xab/0xea
[<ffffffff8100b776>] do_IRQ+0xf4/0x132
[<ffffffff81009956>] ret_from_intr+0x0/0xf
DWARF2 unwinder stuck at ret_from_intr+0x0/0xf
Leftover inexact backtrace:
<IRQ> <EOI> [<ffffffff8159f61d>] start_kernel+0x178/0x2f6
[<ffffffff8159f625>] start_kernel+0x180/0x2f6
[<ffffffff8159f61d>] start_kernel+0x178/0x2f6
[<ffffffff8159f13e>] _sinittext+0x13e/0x142
BUG: spinlock lockup on CPU#0, swapper/0, ffffffff81586140
Call Trace:
[<ffffffff8100a6f9>] show_trace+0x34/0x47
[<ffffffff8100a71e>] dump_stack+0x12/0x17
[<ffffffff811457c8>] _raw_spin_lock+0xca/0xe8
[<ffffffff8104d139>] handle_level_irq+0xb3/0xea
[<ffffffff8100b776>] do_IRQ+0xf4/0x132
[<ffffffff81009956>] ret_from_intr+0x0/0xf
DWARF2 unwinder stuck at ret_from_intr+0x0/0xf
Leftover inexact backtrace:
<IRQ> <EOI> [<ffffffff8159f61d>] start_kernel+0x178/0x2f6
[<ffffffff8159f625>] start_kernel+0x180/0x2f6
[<ffffffff8159f61d>] start_kernel+0x178/0x2f6
[<ffffffff8159f13e>] _sinittext+0x13e/0x142
Thanks
Vivek
On Mon, Nov 20, 2006 at 02:23:35PM -0500, Vivek Goyal wrote:
> On Fri, Nov 10, 2006 at 04:55:48PM +0300, Pavel Emelianov wrote:
> > As the second lock on booth CPUs is taken before checking that
> > this irq is being handled in another processor this may cause
> > a deadlock. This issue is only theoretical.
> >
> > I propose the attached patch to fix booth problems: when trying
> > to handle misrouted IRQ active desc->lock may be unlocked.
> >
> > Please comment.
>
> > --- ./kernel/irq/spurious.c.irqlockup 2006-11-09 11:19:10.000000000 +0300
> > +++ ./kernel/irq/spurious.c 2006-11-10 16:53:38.000000000 +0300
> > @@ -147,7 +147,11 @@ void note_interrupt(unsigned int irq, st
> > if (unlikely(irqfixup)) {
> > /* Don't punish working computers */
> > if ((irqfixup == 2 && irq == 0) || action_ret == IRQ_NONE) {
> > - int ok = misrouted_irq(irq);
> > + int ok;
> > +
> > + spin_unlock(&desc->lock);
> > + ok = misrouted_irq(irq);
> > + spin_lock(&desc->lock);
>
> Hi Pavel,
>
> Till -rc5, I was able to boot a kernel with irqpoll option and in -rc6 I
> can't. It simply hangs. I suspect it is realted to this change. I have yet
> to confirm that. But before that one observation.
>
Hi Pavel,
If I backout your changes, everything works fine. So it looks like that
the problem I am facing is because of your patch but I don't have a logical
explanation yet that why the problem is there. Just realasing a lock
which is not currently acquired should not hang the system?
Thanks
Vivek
On Mon, Nov 20, 2006 at 02:56:52PM -0500, Vivek Goyal wrote:
> On Mon, Nov 20, 2006 at 02:23:35PM -0500, Vivek Goyal wrote:
> > On Fri, Nov 10, 2006 at 04:55:48PM +0300, Pavel Emelianov wrote:
> > > As the second lock on booth CPUs is taken before checking that
> > > this irq is being handled in another processor this may cause
> > > a deadlock. This issue is only theoretical.
> > >
> > > I propose the attached patch to fix booth problems: when trying
> > > to handle misrouted IRQ active desc->lock may be unlocked.
> > >
> > > Please comment.
> >
> > > --- ./kernel/irq/spurious.c.irqlockup 2006-11-09 11:19:10.000000000 +0300
> > > +++ ./kernel/irq/spurious.c 2006-11-10 16:53:38.000000000 +0300
> > > @@ -147,7 +147,11 @@ void note_interrupt(unsigned int irq, st
> > > if (unlikely(irqfixup)) {
> > > /* Don't punish working computers */
> > > if ((irqfixup == 2 && irq == 0) || action_ret == IRQ_NONE) {
> > > - int ok = misrouted_irq(irq);
> > > + int ok;
> > > +
> > > + spin_unlock(&desc->lock);
> > > + ok = misrouted_irq(irq);
> > > + spin_lock(&desc->lock);
> >
> > Hi Pavel,
> >
> > Till -rc5, I was able to boot a kernel with irqpoll option and in -rc6 I
> > can't. It simply hangs. I suspect it is realted to this change. I have yet
> > to confirm that. But before that one observation.
> >
>
> Hi Pavel,
>
> If I backout your changes, everything works fine. So it looks like that
> the problem I am facing is because of your patch but I don't have a logical
> explanation yet that why the problem is there. Just realasing a lock
> which is not currently acquired should not hang the system?
>
Some more data regarding this issue.
For my system it gets locked in following sequence.
handle_level_irq() {
spin_unlock(desc->lock)
......
note_interrupt() {
/* Called without desc->lock held */
spin_unlock(desc->lock)
misrouted_irq()
spin_lock(desc->lock)
}
spin_lock(desc->lock) /* Lockup */
}
So basically problems seems to be due to calling conventions of
note_interrupt(). In few cases we call it with desc->lock held and in other
cases we call it with desc->lock not held.
I guess, note_interrupt() should restore the desc->lock status back to the
state of the lock when function was entered so that caller does not lockup.
Thanks
Vivek
Vivek Goyal wrote:
> On Mon, Nov 20, 2006 at 02:23:35PM -0500, Vivek Goyal wrote:
>> On Fri, Nov 10, 2006 at 04:55:48PM +0300, Pavel Emelianov wrote:
>>> As the second lock on booth CPUs is taken before checking that
>>> this irq is being handled in another processor this may cause
>>> a deadlock. This issue is only theoretical.
>>>
>>> I propose the attached patch to fix booth problems: when trying
>>> to handle misrouted IRQ active desc->lock may be unlocked.
>>>
>>> Please comment.
>>> --- ./kernel/irq/spurious.c.irqlockup 2006-11-09 11:19:10.000000000 +0300
>>> +++ ./kernel/irq/spurious.c 2006-11-10 16:53:38.000000000 +0300
>>> @@ -147,7 +147,11 @@ void note_interrupt(unsigned int irq, st
>>> if (unlikely(irqfixup)) {
>>> /* Don't punish working computers */
>>> if ((irqfixup == 2 && irq == 0) || action_ret == IRQ_NONE) {
>>> - int ok = misrouted_irq(irq);
>>> + int ok;
>>> +
>>> + spin_unlock(&desc->lock);
>>> + ok = misrouted_irq(irq);
>>> + spin_lock(&desc->lock);
>> Hi Pavel,
>>
>> Till -rc5, I was able to boot a kernel with irqpoll option and in -rc6 I
>> can't. It simply hangs. I suspect it is realted to this change. I have yet
>> to confirm that. But before that one observation.
>>
>
> Hi Pavel,
>
> If I backout your changes, everything works fine. So it looks like that
> the problem I am facing is because of your patch but I don't have a logical
> explanation yet that why the problem is there. Just realasing a lock
> which is not currently acquired should not hang the system?
Without this patch my kernel hanged in another place.
I'll look over the interrupt code again. I suspect that
just another fix is required.
> Hi Pavel,
>
> If I backout your changes, everything works fine. So it looks like that
> the problem I am facing is because of your patch but I don't have a logical
> explanation yet that why the problem is there. Just realasing a lock
> which is not currently acquired should not hang the system?
Hm... A simple grep over the code showed that note_interrupt
is called w/o desc->lock in all places but __do_IRQ(). And this
looks like an error at least for the following reason:
note_interrupt() calls __report_bad_irq() and __report_bad_irq()
does require desc->lock to be held. So I suppose that we have
to do spin_lock(&desc->lock) before calling note_interrupt().
I'll prepare a patch in a moment.