by Jesse Brandeburg

[permalink] [raw]

Subject: Re: bisect results of MSI-X related panic (help!)

On Mon, 12 Oct 2009, Tejun Heo wrote:
> Can you please apply the following patch and try to retrigger the
> panic?
>
> diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c
> index c166019..f5a1482 100644
> --- a/kernel/irq/chip.c
> +++ b/kernel/irq/chip.c
> @@ -63,6 +63,9 @@ void dynamic_irq_cleanup(unsigned int irq)
> struct irq_desc *desc = irq_to_desc(irq);
> unsigned long flags;
>
> + printk("XXX dynamic_irq_cleanup() called on %u\n", irq);
> + dump_stack();
> +
> if (!desc) {
> WARN(1, KERN_ERR "Trying to cleanup invalid IRQ%d\n", irq);
> return;

I'm working on it, but now that I've added a bunch of debug including the
above printk, my system panics (with a stack protector canary overwrite)
when loading the first network adapter with 30+ MSI-X vectors. I can boot
single user mode and bring up netconsole, but then as soon as I brought up
the first port with lots of MSI-X vectors, the system hard locks, no panic
message.

I have a bit of a theory that the node = -1 (numa_node) stuff might be
playing some havoc with the code in numa_migrate.c. I'm not sure if that
is contributing, but the code in there doesn't seem written to handle node
= - 1 very well. As in I never see it do an smp_processor_id at the
bottom before accessing the node value.

Not sure if that is relevant, but I wanted to mention it before I went
home.

What next? I made it worse so I guess that is something.

2009-10-15 07:32:57

by Tejun Heo

[permalink] [raw]

Subject: Re: bisect results of MSI-X related panic (help!)

Hello,

Brandeburg, Jesse wrote:
> On Mon, 12 Oct 2009, Tejun Heo wrote:
>> Can you please apply the following patch and try to retrigger the
>> panic?
>>
>> diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c
>> index c166019..f5a1482 100644
>> --- a/kernel/irq/chip.c
>> +++ b/kernel/irq/chip.c
>> @@ -63,6 +63,9 @@ void dynamic_irq_cleanup(unsigned int irq)
>> struct irq_desc *desc = irq_to_desc(irq);
>> unsigned long flags;
>>
>> + printk("XXX dynamic_irq_cleanup() called on %u\n", irq);
>> + dump_stack();
>> +
>> if (!desc) {
>> WARN(1, KERN_ERR "Trying to cleanup invalid IRQ%d\n", irq);
>> return;
>
> I'm working on it, but now that I've added a bunch of debug including the
> above printk, my system panics (with a stack protector canary overwrite)
> when loading the first network adapter with 30+ MSI-X vectors. I can boot
> single user mode and bring up netconsole, but then as soon as I brought up
> the first port with lots of MSI-X vectors, the system hard locks, no panic
> message.
>
> I have a bit of a theory that the node = -1 (numa_node) stuff might be
> playing some havoc with the code in numa_migrate.c. I'm not sure if that
> is contributing, but the code in there doesn't seem written to handle node
> = - 1 very well. As in I never see it do an smp_processor_id at the
> bottom before accessing the node value.
>
> Not sure if that is relevant, but I wanted to mention it before I went
> home.
>
> What next? I made it worse so I guess that is something.

I don't know. At this point, I can't think of anything other than
sprinkling printks and dump_stacks around. :-(

Thanks.

--
tejun