Message-ID: <487F45C0.2080201@csr.com>
Date: Thu, 17 Jul 2008 14:14:40 +0100
From: David Vrabel <david.vrabel@csr.com>
User-Agent: Thunderbird 2.0.0.14 (X11/20080505)
MIME-Version: 1.0
To: Matthew Wilcox <matthew@wil.cx>
CC: Jesse Barnes <jbarnes@virtuousgeek.org>,
       Kernel development list <linux-kernel@vger.kernel.org>,
       Ingo Molnar <mingo@elte.hu>, Thomas Gleixner <tglx@linutronix.de>
Subject: Re: PCI: MSI interrupts masked using prohibited method
References: <4860D09D.4060801@csr.com> <4864DA54.6000205@csr.com> <200806271007.26248.jbarnes@virtuousgeek.org> <200807161243.03360.jbarnes@virtuousgeek.org> <20080716195853.GL25255@parisc-linux.org>
In-Reply-To: <20080716195853.GL25255@parisc-linux.org>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 4456
Lines: 92

Matthew Wilcox wrote:
> On Wed, Jul 16, 2008 at 12:43:03PM -0700, Jesse Barnes wrote:
>> On Friday, June 27, 2008 10:07 am Jesse Barnes wrote:
>>> On Friday, June 27, 2008 5:17 am David Vrabel wrote:
>>>> Jesse Barnes wrote:
>>>>> On Tuesday, June 24, 2008 3:46 am David Vrabel wrote:
>>>>>> PCI MSI interrupts are masked and unmasked using a method (by writing
>>>>>> the MSI Enable capability bit) that is prohibited by the PCI
>>>>>> specification.
>>>>> Yeah, it's probably quite a bit slower too (I assume you're talking
>>>>> about io_apic_64's msi_mask_irq).  Seems like masking this at the
>>>>> ioapic level would make more sense anyway...
>>>>>
>>>>>> This behaviour can cause missed interrupts with some devices if the
>>>>>> interrupt is asserted by the hardware while MSI is disabled.
>>>>>>
>>>>>> I believe the interrupt should be masked/unmasked on the interrupt
>>>>>> controller (the APIC on x86, for example).   I'm going to test this
>>>>>> now and see if it works.
>>>> After further research it seems that MSI interrupts aren't routed via
>>>> the IO-APIC, so this cannot be done.
>>>>
>>>> I think the only solution is to not perform any sort of masking and rely
>>>> on the device driver being able to handle this.
>>> On x86, they're targetted at the LAPIC block (see section 8 of the IA SDM);
>>> maybe we could modify the message address or data such that it won't
>>> generate an interrupt instead?  I think this latest approach is correct in
>>> the sense that both the system and drivers have to take care that
>>>   1) we don't miss interrupts, and
>>>   2) we don't generate spurious unhandled interrupts (as might happen if we
>>> disable MSI and the device generates a legacy IRQ on a different vector).
>>>
>>> But it looks like the real problem is in the system interrupt code that
>>> handles MSIs.  We should only be disabling MSIs using the capability bit at
>>> device enable or disable time, not during the normal course of interrupt
>>> handling, since if we do we may miss device interrupts or have them routed
>>> to the wrong (legacy) vector.
>>>
>>> Cc'ing Ingo & Thomas since they know the core interrupt code pretty well.
>> Ingo or Matthew, any ideas about this?  The fundamental issue is that if we go 
>> poke at a device's MSI cap bits during interrupt handling, the device may 
>> start using regular IRQs instead, potentially on a different vector.  It 
>> would be good if we could come up with a better way of masking MSIs during 
>> handling...
> 
> OK, I didn't see the initial report, but I've looked at the PCI MSI
> masking a little bit due to the multiple MSI work.
> 
> The first thing is that some MSI implementations permit masking on the
> PCI device.  That's handled through the
> 
>                 if (entry->msi_attrib.maskbit) {
> 
> part of msi_set_mask_bits() in drivers/pci/msi.c.
> 
> David's clearly talking about devices which don't support mask bits.
> For these devices, we call
>                         msi_set_enable(entry->dev, !flag);
> 
> which turns off the PCI_MSI_FLAGS_ENABLE bit.  I'm not sure I see the
> bit in the PCI spec that says devices are then allowed to ignore the
> Interrupt Disable bit in the device control register, but I concede such
> devices may be out there.

>From the PCI spec:

"This bit disables the device/function from asserting INTx#. A value of
0 enables the assertion of its INTx# signal. A value of 1 disables the
assertion of its INTx# signal. This bit?s state after RST# is 0. Refer
to Section 6.8.1.3 for control of MSI."

So the interrupt disable bit is only for the line interrupts, and not MSI.

> We have some infrastructure for resending IRQs, so we could soft-mask an
> MSI and resend it when the irq is unmasked, if it has triggered.

Is this really necessary?  Any PCI device with MSI that doesn't have the
hardware MSI mask bits must have some sort of device specific
handshaking for managing when MSIs can be generated.

Regardless, doesn't __do_IRQ() handle this already anyway?

David
-- 
David Vrabel, Senior Software Engineer, Drivers
CSR, Churchill House, Cambridge Business Park,  Tel: +44 (0)1223 692562
Cowley Road, Cambridge, CB4 0WZ                 http://www.csr.com/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/