Has anybody done any work, or put any thought, into MSI support?
Would things massively break if I set up MSI manually in the driver?
I heard rumblings on lkml that Intel has done some work internally w/
MSI support in Linux, but that doesn't help me much without further
details ;-)
Jeff
On Mon, May 12, 2003 at 12:32:49PM -0400, Jeff Garzik wrote:
> Has anybody done any work, or put any thought, into MSI support?
Work -- no. Thought? A little. Seems to me that MSIs need to be treated
as a third form of interrupts (level/edge/message). The address that
the MSI will write to is clearly architecture dependent (may even be
irq-controller-dependent, depending on your architecture). request_irq()
is an insufficient function to deal with this -- request_msi() may be
needed instead. It'll need to return an address to pass to the card.
(We need a mechanism to decide whether it's a 32-bit or 64-bit address).
Oh, and don't make this too PCI-specific -- native PARISC interrupts
are MSI and you can see how handled it in arch/parisc/kernel/irq.c.
> Would things massively break if I set up MSI manually in the driver?
Might do, might not.
--
"It's not Hollywood. War is real, war is primarily not about defeat or
victory, it is about death. I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk
On Mon, May 12, 2003 at 05:53:31PM +0100, Matthew Wilcox wrote:
> On Mon, May 12, 2003 at 12:32:49PM -0400, Jeff Garzik wrote:
> > Has anybody done any work, or put any thought, into MSI support?
>
> Work -- no. Thought? A little. Seems to me that MSIs need to be treated
> as a third form of interrupts (level/edge/message). The address that
> the MSI will write to is clearly architecture dependent (may even be
> irq-controller-dependent, depending on your architecture). request_irq()
> is an insufficient function to deal with this -- request_msi() may be
> needed instead. It'll need to return an address to pass to the card.
> (We need a mechanism to decide whether it's a 32-bit or 64-bit address).
I've also done some thought for PPC440xx's PCI MSI support. It isn't
strictly necessary to have a new request_msi() if the kernel "does
the right thing". request_irq() already hooks using an interrupt
value that is virtual on many platforms. In that case, the PCI
subsystem would only need to provide an interface to provide
the architecture/platform specific inbound MSI location. The PCI
subsystem would then find all MSI capable PCI devices, and assign
the appropriate number of unique messages and inbound MSI address
to each device via the speced PCI MSI interface. The PCI subsystem
would also be responsible for maintaining a correspondence between
virtual Linux interrupt values and MSI values.
Software specific to the PCI MSI capable "Northbridge", will then
route general MSI interrupt events to some PCI subsystem helper
functions to verify which MSI has occurred and thus which Linux
virtual interrupt.
Perhaps request_irq() just needs to be explicitly abstracted if
an unsigned int is not sufficient for the entire message space or
if we want messages unique only on a per-space basis i.e. PCI MSIs
can be dups of RapidIO MSIs, etc.
> Oh, and don't make this too PCI-specific -- native PARISC interrupts
> are MSI and you can see how handled it in arch/parisc/kernel/irq.c.
FWIW, another interconnect that natively uses MSI is RapidIO. It's
all in-band doorbell messages, no out-of-band discrete interrupts like
PCI.
Regards,
--
Matt Porter
[email protected]
On Mon, May 12, 2003 at 10:43:00AM -0700, Matt Porter wrote:
> I've also done some thought for PPC440xx's PCI MSI support. It isn't
> strictly necessary to have a new request_msi() if the kernel "does
> the right thing". request_irq() already hooks using an interrupt
> value that is virtual on many platforms.
Yes, but ideally this kludge would go away...
> In that case, the PCI
> subsystem would only need to provide an interface to provide
> the architecture/platform specific inbound MSI location. The PCI
> subsystem would then find all MSI capable PCI devices, and assign
> the appropriate number of unique messages and inbound MSI address
> to each device via the speced PCI MSI interface. The PCI subsystem
> would also be responsible for maintaining a correspondence between
> virtual Linux interrupt values and MSI values.
>
> Software specific to the PCI MSI capable "Northbridge", will then
> route general MSI interrupt events to some PCI subsystem helper
> functions to verify which MSI has occurred and thus which Linux
> virtual interrupt.
That sounds like a lot of overhead. In particular it means we keep
converting to and from `virtual IRQs'. I would hope the MSI work would
allow us to tie in at a lower level than virtual interrupts. I was
thinking an interface would look something like:
void *request_msi(struct device *dev,
irqreturn_t (*handler)(int, void *, struct pt_regs *),
unsigned long irqflags,
void *dev_id)
You need a struct device to figure out which interrupt controller it
needs.
--
"It's not Hollywood. War is real, war is primarily not about defeat or
victory, it is about death. I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk
From: Matthew Wilcox <[email protected]>
Date: Mon, 12 May 2003 17:53:31 +0100
On Mon, May 12, 2003 at 12:32:49PM -0400, Jeff Garzik wrote:
> Has anybody done any work, or put any thought, into MSI support?
Work -- no. Thought? A little. Seems to me that MSIs need to be
treated as a third form of interrupts (level/edge/message).
The fact that Alpha already supports them pretty much transparently
suggests that the thing to do might very well be "nothing" :-)
To be honest, MSIs are very similar to how interrupts work on sparc64,
in that each device generates a unique interrupt cookie. The only
different is the size of this cookie, MSIs are larger than sparc64's.
On Mon, May 12, 2003 at 07:20:23PM +0100, Matthew Wilcox wrote:
> On Mon, May 12, 2003 at 10:43:00AM -0700, Matt Porter wrote:
> > I've also done some thought for PPC440xx's PCI MSI support. It isn't
> > strictly necessary to have a new request_msi() if the kernel "does
> > the right thing". request_irq() already hooks using an interrupt
> > value that is virtual on many platforms.
>
> Yes, but ideally this kludge would go away...
Well, with respect to MSI, yes it could. Interrupt values are
still a Linux fabrication regardless on many platforms with
cascaded PICs.
> > In that case, the PCI
> > subsystem would only need to provide an interface to provide
> > the architecture/platform specific inbound MSI location. The PCI
> > subsystem would then find all MSI capable PCI devices, and assign
> > the appropriate number of unique messages and inbound MSI address
> > to each device via the speced PCI MSI interface. The PCI subsystem
> > would also be responsible for maintaining a correspondence between
> > virtual Linux interrupt values and MSI values.
> >
> > Software specific to the PCI MSI capable "Northbridge", will then
> > route general MSI interrupt events to some PCI subsystem helper
> > functions to verify which MSI has occurred and thus which Linux
> > virtual interrupt.
>
> That sounds like a lot of overhead. In particular it means we keep
> converting to and from `virtual IRQs'. I would hope the MSI work would
> allow us to tie in at a lower level than virtual interrupts. I was
> thinking an interface would look something like:
Yes, it's a bit. Just pointing out that one *could* do it that way,
not really that we *should* do it that way. I would personally
prefer to see native msi support. However, most of the functionality
I mentioned are things the PCI subsystem needs to do specific to
PCI MSI anyway.
> void *request_msi(struct device *dev,
> irqreturn_t (*handler)(int, void *, struct pt_regs *),
> unsigned long irqflags,
> void *dev_id)
>
> You need a struct device to figure out which interrupt controller it
> needs.
request_msi() needs an additional parameter to specify which MSI
it is hooking. A device can implement many messages in order to
clarify which one of many events on a device has occurred. It
may be desired to hook a separate handler for each of those to
avoid another read of a status register.
Regards,
--
Matt Porter
[email protected]
On Mon, May 12, 2003 at 11:48:51AM -0700, Matt Porter wrote:
> request_msi() needs an additional parameter to specify which MSI
> it is hooking. A device can implement many messages in order to
> clarify which one of many events on a device has occurred. It
> may be desired to hook a separate handler for each of those to
> avoid another read of a status register.
Assuming that platform specific PCI setup does reasonable real to virtual
IRQ mapping, request_msi() is not needed. We can use pdev->irq
as "base" vector and MSI message number as offset. Alpha works this
way, BTW.
I think of something like this:
/**
* pci_using_msi - is this PCI device configured to use MSI?
* @dev: PCI device structure of device being queried
*
* Tells whether or not a PCI device is configured to use Message Signaled
* Interrupts. Returns number of allocated MSI messages, else 0.
*/
int
pci_using_msi(struct pci_dev *dev)
{
int msi = pci_find_capability(dev, PCI_CAP_ID_MSI);
u8 msgctl;
if (!msi || !dev->irq)
return 0;
pci_read_config_byte(dev, msi + PCI_MSI_FLAGS, &msgctl);
if (!(msgctl & PCI_MSI_FLAGS_ENABLE))
return 0;
return 1 << ((msgctl >> 4) & 7); /* # of messages allocated */
}
So that MSI-aware driver can do
nummsgs = pci_using_msi(dev);
if (!nummsgs)
goto no_msi;
for (msg = 0; msg < nummsgs; msg++) {
...
request_irq(dev->irq + msg, ...);
}
Ivan.