Hi,
I am wondering how to handle this:
Does the kernel somehow handle MSI message readback from the PCI config space
for the MSI message as described in 6.8.1 of the PCI Bus specification 2.3 or
does a device specific driver have to read the Message Address and Data ?
To put it short: i am wondering how i should read the MSI messages.
Thanks,
Manu
On Sat, Dec 13, 2008 at 01:38:24PM +0400, Manu Abraham wrote:
> Hi,
>
> I am wondering how to handle this:
>
> Does the kernel somehow handle MSI message readback from the PCI config space
> for the MSI message as described in 6.8.1 of the PCI Bus specification 2.3 or
> does a device specific driver have to read the Message Address and Data ?
>
> To put it short: i am wondering how i should read the MSI messages.
The "message" is actually mapped to an the interrupt vector by the core
generic interrupt handling code in the kernel.
A "GSI" (Generic Sys Interrupt?) is associated with each entry in
the MSI-X table. Driver then calls request_irq() to bind an interrupt
handler to each GSI. So the driver never directly sees the "message".
hth,
grant
Grant Grundler wrote:
> On Sat, Dec 13, 2008 at 01:38:24PM +0400, Manu Abraham wrote:
>> Hi,
>>
>> I am wondering how to handle this:
>>
>> Does the kernel somehow handle MSI message readback from the PCI config space
>> for the MSI message as described in 6.8.1 of the PCI Bus specification 2.3 or
>> does a device specific driver have to read the Message Address and Data ?
>>
>> To put it short: i am wondering how i should read the MSI messages.
>
> The "message" is actually mapped to an the interrupt vector by the core
> generic interrupt handling code in the kernel.
>
> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
> the MSI-X table. Driver then calls request_irq() to bind an interrupt
> handler to each GSI. So the driver never directly sees the "message".
Oh, you mean the array of irq_handlers in the MSI-X table should
correspond to a particular message ?
I remember sometime back somebody stated that one might not get
the requested number of handlers what you requested.
In such a case, if the irq_handlers are mapped in a static fashion, one
might not get any interrupt callbacks for a specific functionality of
the hardware. Or is there something missing in what i understood ?
Regards,
Manu
On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
...
> > A "GSI" (Generic Sys Interrupt?) is associated with each entry in
> > the MSI-X table. Driver then calls request_irq() to bind an interrupt
> > handler to each GSI. So the driver never directly sees the "message".
>
> Oh, you mean the array of irq_handlers in the MSI-X table should
> correspond to a particular message ?
No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
is associated with the parameters passed to each call of request_irq().
Pass in unique private data to each call of request_irq() and the driver
can determine which message was delivered when the ISR gets called.
(ISR == Interrupt Service Routine, aka driver IRQ handler)
> I remember sometime back somebody stated that one might not get
> the requested number of handlers what you requested.
Correct. The limit is defined by the number of GSI's available
from the HW.
> In such a case, if the irq_handlers are mapped in a static fashion, one
> might not get any interrupt callbacks for a specific functionality of
> the hardware. Or is there something missing in what i understood ?
Correct.
grant
Grant Grundler wrote:
> A "GSI" (Generic Sys Interrupt?)
Should be "Global System Interrupt", as described in Section 5.2.12 of
ACPIspec3.0b ;-)
Grant Grundler wrote:
> On Sat, Dec 13, 2008 at 01:38:24PM +0400, Manu Abraham wrote:
>> Hi,
>>
>> I am wondering how to handle this:
>>
>> Does the kernel somehow handle MSI message readback from the PCI config space
>> for the MSI message as described in 6.8.1 of the PCI Bus specification 2.3 or
>> does a device specific driver have to read the Message Address and Data ?
>>
>> To put it short: i am wondering how i should read the MSI messages.
>
> The "message" is actually mapped to an the interrupt vector by the core
> generic interrupt handling code in the kernel.
>
> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
> the MSI-X table. Driver then calls request_irq() to bind an interrupt
> handler to each GSI. So the driver never directly sees the "message".
>
I think "GSI (Global System Interrupt)" is for identifying the I/O
APIC pin among multiple I/O APICs. Maybe you wanted to mean the
interrupt number managed by kernel (frequently called "IRQ")?
Or am I misunderstanding something?
Thanks,
Kenji Kaneshige
Grant Grundler wrote:
> On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
> ...
>>> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
>>> the MSI-X table. Driver then calls request_irq() to bind an interrupt
>>> handler to each GSI. So the driver never directly sees the "message".
>> Oh, you mean the array of irq_handlers in the MSI-X table should
>> correspond to a particular message ?
>
> No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
> is associated with the parameters passed to each call of request_irq().
> Pass in unique private data to each call of request_irq() and the driver
> can determine which message was delivered when the ISR gets called.
> (ISR == Interrupt Service Routine, aka driver IRQ handler)
Is it possible that even when the config space says that multiple messages
are supported, you cannot enable MSI-X ?
To explain what i mean here, from the logs in the 3 modes of operation
MSI-X, MSI, INTx, when i am requesting MSI-X mode, i find that the MSI-X
mode doesn't get enabled. Is this something that's supposed to be ?
The codebase that generates the logs is http://jusst.de/hg/saa716x
the module that causes it is the saa716x module @ linux/drivers/media/dvb/saa716x
Regards,
Manu
------------- with MSI-X -------------
saa716x_pci_init (0): found a NEMO reference board PCIe card
ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
PCI: Setting latency timer of device 0000:05:00.0 to 64
saa716x_request_irq (0): Using MSI-X mode
saa716x_enable_msix (0): MSI-X request failed
saa716x_request_irq (0): INT-A Mode
SAA7160 Rev 1 [1131:0000], irq: 19,
mmio: 0xf9480000
SAA7160 64Bit, MSI Disabled, MSI-X=32 msgs
saa716x_getbootscript_setup (0): Domain 0: Clk PSS <0x218> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 1: Clk DCS <0x21c> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 2: Clk SPI <0x220> Divider: 0x7f825 --> N=1, M=5, freq=500
saa716x_getbootscript_setup (0): Domain 3: Clk I2C/Boot <0x224> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 4: Clk PHI <0x228> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 5: Clk VI0 <0x22c> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 6: Clk VI1 <0x230> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 7: Clk FGPI0 <0x234> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 8: Clk FGPI1 <0x238> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 9: Clk FGPI2 <0x23c> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 10: Clk FGPI3 <0x240> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 11: Clk AI0 <0x244> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 12: Clk AI1 <0x248> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 13: Clk Phy <0x24c> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_core_boot (0): Using External Boot from config
saa716x_hybrid_pci_probe (0): SAA716x Core Boot Success
saa716x_msi_init (0): Initializing MSI ..
saa716x_msi_init (0): Interrupts ena_l <00> ena_h <00> sta_l <00> sta_h <00>
saa716x_jetpack_init (0): SAA7160 Jetpack Successfully initialized
saa716x_i2c_init (0): Initializing SAA7160 I2C Core
saa716x_i2c_init (0): Initializing adapter (0) SAA716x I2C Core 0
saa716x_i2c_init (0): Initializing Adapter (0) SAA716x I2C Core 0 @ 100k
saa716x_add_irqvector (0): Adding Vector 49 <I2CINT_0>
saa716x_add_irqvector (0): Interrupts ena_l <00> ena_h <20000>
saa716x_i2c_init (0): Initializing adapter (1) SAA716x I2C Core 1
saa716x_hybrid_pci_irq (0): MSI STAT L=<00> H=<20000>, CTL L=<00> H=<20000>
saa716x_msi_event: MSI event <I2CINT_0> Bus(0) I2C event: Status=<Idle> --> Stat=<00> Mask=<c7>
saa716x_i2c_init (0): Initializing Adapter (1) SAA716x I2C Core 1 @ 100k
saa716x_add_irqvector (0): Adding Vector 50 <I2CINT_1>
saa716x_add_irqvector (0): Interrupts ena_l <00> ena_h <60000>
saa716x_i2c_init (0): SAA7160 I2C Core succesfully initialized
saa716x_hybrid_pci_irq (0): MSI STAT L=<00> H=<40000>, CTL L=<00> H=<60000>
saa716x_msi_event: MSI event <I2CINT_1> Bus(1) I2C event: Status=<Idle> --> Stat=<00> Mask=<c7>
saa716x_dvb_init (0): dvb_register_adapter
DVB: registering new adapter (SAA716x dvb adapter)
saa716x_dvb_init (0): dvb_dmx_init
saa716x_dvb_init (0): dvb_dmxdev_init
saa716x_dvb_init (0): Frontend Init
saa716x_frontend_power (0): Adapter (0) Power ON
saa716x_frontend_reset (0): Adapter (0) RESET
saa716x_nemo_frontend_attach (0): Adapter (0) SAA716x frontend Init
saa716x_nemo_frontend_attach (0): Adapter (0) Device ID=00
------------- with MSI -------------
saa716x_pci_init (0): found a NEMO reference board PCIe card
ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
PCI: Setting latency timer of device 0000:05:00.0 to 64
saa716x_request_irq (0): Using MSI mode
SAA7160 Rev 1 [1131:0000], irq: 215,
mmio: 0xf9480000
SAA7160 64Bit, MSI Enabled, MSI-X=32 msgs
saa716x_getbootscript_setup (0): Domain 0: Clk PSS <0x218> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 1: Clk DCS <0x21c> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 2: Clk SPI <0x220> Divider: 0x7f825 --> N=1, M=5, freq=500
saa716x_getbootscript_setup (0): Domain 3: Clk I2C/Boot <0x224> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 4: Clk PHI <0x228> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 5: Clk VI0 <0x22c> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 6: Clk VI1 <0x230> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 7: Clk FGPI0 <0x234> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 8: Clk FGPI1 <0x238> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 9: Clk FGPI2 <0x23c> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 10: Clk FGPI3 <0x240> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 11: Clk AI0 <0x244> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 12: Clk AI1 <0x248> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 13: Clk Phy <0x24c> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_core_boot (0): Using External Boot from config
saa716x_hybrid_pci_probe (0): SAA716x Core Boot Success
saa716x_msi_init (0): Initializing MSI ..
saa716x_msi_init (0): Interrupts ena_l <00> ena_h <00> sta_l <00> sta_h <00>
saa716x_jetpack_init (0): SAA7160 Jetpack Successfully initialized
saa716x_i2c_init (0): Initializing SAA7160 I2C Core
saa716x_i2c_init (0): Initializing adapter (0) SAA716x I2C Core 0
saa716x_i2c_init (0): Initializing Adapter (0) SAA716x I2C Core 0 @ 100k
saa716x_add_irqvector (0): Adding Vector 49 <I2CINT_0>
saa716x_add_irqvector (0): Interrupts ena_l <00> ena_h <20000>
saa716x_i2c_init (0): Initializing adapter (1) SAA716x I2C Core 1
irq 64, desc: c0483e80, depth: 1, count: 0, unhandled: 0
->handle_irq(): c01550f9, handle_bad_irq+0x0/0x1a5
->chip(): c0456030, no_irq_chip+0x0/0x40
->action(): 00000000
IRQ_DISABLED set
unexpected IRQ trap at vector 40
saa716x_i2c_init (0): Initializing Adapter (1) SAA716x I2C Core 1 @ 100k
saa716x_add_irqvector (0): Adding Vector 50 <I2CINT_1>
saa716x_add_irqvector (0): Interrupts ena_l <00> ena_h <60000>
saa716x_i2c_init (0): SAA7160 I2C Core succesfully initialized
irq 64, desc: c0483e80, depth: 1, count: 0, unhandled: 0
->handle_irq(): c01550f9, handle_bad_irq+0x0/0x1a5
->chip(): c0456030, no_irq_chip+0x0/0x40
->action(): 00000000
IRQ_DISABLED set
unexpected IRQ trap at vector 40
saa716x_dvb_init (0): dvb_register_adapter
DVB: registering new adapter (SAA716x dvb adapter)
saa716x_dvb_init (0): dvb_dmx_init
saa716x_dvb_init (0): dvb_dmxdev_init
saa716x_dvb_init (0): Frontend Init
saa716x_frontend_power (0): Adapter (0) Power ON
saa716x_frontend_reset (0): Adapter (0) RESET
saa716x_nemo_frontend_attach (0): Adapter (0) SAA716x frontend Init
saa716x_nemo_frontend_attach (0): Adapter (0) Device ID=00
-------------with INTx -------------
saa716x_pci_init (0): found a NEMO reference board PCIe card
ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
PCI: Setting latency timer of device 0000:05:00.0 to 64
SAA7160 Rev 1 [1131:0000], irq: 19,
mmio: 0xf9480000
SAA7160 64Bit, MSI Disabled, MSI-X=32 msgs
saa716x_getbootscript_setup (0): Domain 0: Clk PSS <0x218> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 1: Clk DCS <0x21c> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 2: Clk SPI <0x220> Divider: 0x7f825 --> N=1, M=5, freq=500
saa716x_getbootscript_setup (0): Domain 3: Clk I2C/Boot <0x224> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 4: Clk PHI <0x228> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_getbootscript_setup (0): Domain 5: Clk VI0 <0x22c> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 6: Clk VI1 <0x230> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 7: Clk FGPI0 <0x234> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 8: Clk FGPI1 <0x238> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 9: Clk FGPI2 <0x23c> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 10: Clk FGPI3 <0x240> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 11: Clk AI0 <0x244> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 12: Clk AI1 <0x248> Divider: 0x7f84d --> N=1, M=10, freq=250
saa716x_getbootscript_setup (0): Domain 13: Clk Phy <0x24c> Divider: 0x7f80d --> N=1, M=2, freq=1250
saa716x_core_boot (0): Using External Boot from config
saa716x_hybrid_pci_probe (0): SAA716x Core Boot Success
saa716x_msi_init (0): Initializing MSI ..
saa716x_msi_init (0): Interrupts ena_l <00> ena_h <00> sta_l <00> sta_h <00>
saa716x_jetpack_init (0): SAA7160 Jetpack Successfully initialized
saa716x_i2c_init (0): Initializing SAA7160 I2C Core
saa716x_i2c_init (0): Initializing adapter (0) SAA716x I2C Core 0
saa716x_i2c_init (0): Initializing Adapter (0) SAA716x I2C Core 0 @ 100k
saa716x_add_irqvector (0): Adding Vector 49 <I2CINT_0>
saa716x_add_irqvector (0): Interrupts ena_l <00> ena_h <20000>
saa716x_i2c_init (0): Initializing adapter (1) SAA716x I2C Core 1
saa716x_hybrid_pci_irq (0): MSI STAT L=<00> H=<20000>, CTL L=<00> H=<20000>
saa716x_msi_event: MSI event <I2CINT_0> Bus(0) I2C event: Status=<Idle> --> Stat=<00> Mask=<c7>
saa716x_i2c_init (0): Initializing Adapter (1) SAA716x I2C Core 1 @ 100k
saa716x_add_irqvector (0): Adding Vector 50 <I2CINT_1>
saa716x_add_irqvector (0): Interrupts ena_l <00> ena_h <60000>
saa716x_i2c_init (0): SAA7160 I2C Core succesfully initialized
saa716x_hybrid_pci_irq (0): MSI STAT L=<00> H=<40000>, CTL L=<00> H=<60000>
saa716x_msi_event: MSI event <I2CINT_1> Bus(1) I2C event: Status=<Idle> --> Stat=<00> Mask=<c7>
saa716x_dvb_init (0): dvb_register_adapter
DVB: registering new adapter (SAA716x dvb adapter)
saa716x_dvb_init (0): dvb_dmx_init
saa716x_dvb_init (0): dvb_dmxdev_init
saa716x_dvb_init (0): Frontend Init
saa716x_frontend_power (0): Adapter (0) Power ON
saa716x_frontend_reset (0): Adapter (0) RESET
saa716x_nemo_frontend_attach (0): Adapter (0) SAA716x frontend Init
saa716x_nemo_frontend_attach (0): Adapter (0) Device ID=00
On Sun, 2008-12-21 at 01:13 +0400, Manu Abraham wrote:
> Grant Grundler wrote:
> > On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
> > ...
> >>> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
> >>> the MSI-X table. Driver then calls request_irq() to bind an interrupt
> >>> handler to each GSI. So the driver never directly sees the "message".
> >> Oh, you mean the array of irq_handlers in the MSI-X table should
> >> correspond to a particular message ?
> >
> > No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
> > is associated with the parameters passed to each call of request_irq().
> > Pass in unique private data to each call of request_irq() and the driver
> > can determine which message was delivered when the ISR gets called.
> > (ISR == Interrupt Service Routine, aka driver IRQ handler)
>
>
> Is it possible that even when the config space says that multiple messages
> are supported, you cannot enable MSI-X ?
Yes, absolutely. Have a look at pci_msi_check_device() for starters. MSI
can be disabled globally, or per-device by a quirk, the bridges above
your device might not support MSI and the arch code may prevent
MSI/MSI-X for some reason (ie. platform doesn't support it).
There's also a check in pci_enable_msix() to make sure the entries you
pass in are valid.
> ------------- with MSI-X -------------
>
> saa716x_pci_init (0): found a NEMO reference board PCIe card
> ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
> PCI: Setting latency timer of device 0000:05:00.0 to 64
> saa716x_request_irq (0): Using MSI-X mode
> saa716x_enable_msix (0): MSI-X request failed
For starters you should print the error code returned by
pci_enable_msix() - unfortunately it returns EINVAL for many different
reasons, but it will narrow it down a bit.
cheers
--
Michael Ellerman
OzLabs, IBM Australia Development Lab
wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)
We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person
Michael Ellerman wrote:
> On Sun, 2008-12-21 at 01:13 +0400, Manu Abraham wrote:
>> Grant Grundler wrote:
>>> On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
>>> ...
>>>>> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
>>>>> the MSI-X table. Driver then calls request_irq() to bind an interrupt
>>>>> handler to each GSI. So the driver never directly sees the "message".
>>>> Oh, you mean the array of irq_handlers in the MSI-X table should
>>>> correspond to a particular message ?
>>> No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
>>> is associated with the parameters passed to each call of request_irq().
>>> Pass in unique private data to each call of request_irq() and the driver
>>> can determine which message was delivered when the ISR gets called.
>>> (ISR == Interrupt Service Routine, aka driver IRQ handler)
>>
>> Is it possible that even when the config space says that multiple messages
>> are supported, you cannot enable MSI-X ?
>
> Yes, absolutely. Have a look at pci_msi_check_device() for starters. MSI
> can be disabled globally, or per-device by a quirk, the bridges above
> your device might not support MSI and the arch code may prevent
> MSI/MSI-X for some reason (ie. platform doesn't support it).
>
> There's also a check in pci_enable_msix() to make sure the entries you
> pass in are valid.
>
>> ------------- with MSI-X -------------
>>
>> saa716x_pci_init (0): found a NEMO reference board PCIe card
>> ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
>> PCI: Setting latency timer of device 0000:05:00.0 to 64
>> saa716x_request_irq (0): Using MSI-X mode
>> saa716x_enable_msix (0): MSI-X request failed
>
> For starters you should print the error code returned by
> pci_enable_msix() - unfortunately it returns EINVAL for many different
> reasons, but it will narrow it down a bit.
It does return -22
Regards,
Manu
saa716x_request_irq (0): Using MSI-X mode
saa716x_enable_msix (0): MSI-X request failed <-22>
saa716x_request_irq (0): INT-A Mode
SAA7160 Rev 1 [1131:0000], irq: 19,
mmio: 0xf9480000
SAA7160 64Bit, MSI Disabled, MSI-X=32 msgs
Michael Ellerman wrote:
> On Sun, 2008-12-21 at 01:13 +0400, Manu Abraham wrote:
>> Grant Grundler wrote:
>>> On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
>>> ...
>>>>> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
>>>>> the MSI-X table. Driver then calls request_irq() to bind an interrupt
>>>>> handler to each GSI. So the driver never directly sees the "message".
>>>> Oh, you mean the array of irq_handlers in the MSI-X table should
>>>> correspond to a particular message ?
>>> No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
>>> is associated with the parameters passed to each call of request_irq().
>>> Pass in unique private data to each call of request_irq() and the driver
>>> can determine which message was delivered when the ISR gets called.
>>> (ISR == Interrupt Service Routine, aka driver IRQ handler)
>>
>> Is it possible that even when the config space says that multiple messages
>> are supported, you cannot enable MSI-X ?
>
> Yes, absolutely. Have a look at pci_msi_check_device() for starters. MSI
> can be disabled globally, or per-device by a quirk, the bridges above
> your device might not support MSI and the arch code may prevent
> MSI/MSI-X for some reason (ie. platform doesn't support it).
It seems that the bridge does support MSI. It looks like my video device is
using MSI.
Regards,
Manu
pci 0000:01:00.0: Boot video device
PCI: Setting latency timer of device 0000:00:01.0 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:01.0:pcie00]
PCI: Setting latency timer of device 0000:00:1c.0 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.0:pcie00]
Allocate Port Service[0000:00:1c.0:pcie02]
PCI: Setting latency timer of device 0000:00:1c.1 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.1:pcie00]
Allocate Port Service[0000:00:1c.1:pcie02]
PCI: Setting latency timer of device 0000:00:1c.2 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.2:pcie00]
Allocate Port Service[0000:00:1c.2:pcie02]
PCI: Setting latency timer of device 0000:00:1c.3 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.3:pcie00]
Allocate Port Service[0000:00:1c.3:pcie02]
PCI: Setting latency timer of device 0000:00:1c.4 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.4:pcie00]
Allocate Port Service[0000:00:1c.4:pcie02]
vesafb: framebuffer at 0x91000000, mapped to 0xf8880000, using 10240k, total 14336k
vesafb: mode is 1280x1024x16, linelength=2560, pages=1
04:00.0 Multimedia controller: Philips Semiconductors Device 7160 (rev 02)
Subsystem: Technotrend Systemtechnik GmbH Device 3009
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at 93b00000 (64-bit, non-prefetchable) [size=1M]
Capabilities: <access denied>
05:00.0 Multimedia controller: Philips Semiconductors Device 7160 (rev 01)
Subsystem: Philips Semiconductors Device 0000
Flags: bus master, fast devsel, latency 0, IRQ 19
Memory at 93a00000 (64-bit, non-prefetchable) [size=1M]
Capabilities: <access denied>
Kernel driver in use: SAA716x Hybrid
On Mon, 2008-12-22 at 02:56 +0400, Manu Abraham wrote:
> Michael Ellerman wrote:
> > On Sun, 2008-12-21 at 01:13 +0400, Manu Abraham wrote:
> >> Grant Grundler wrote:
> >>> On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
> >>> ...
> >>>>> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
> >>>>> the MSI-X table. Driver then calls request_irq() to bind an interrupt
> >>>>> handler to each GSI. So the driver never directly sees the "message".
> >>>> Oh, you mean the array of irq_handlers in the MSI-X table should
> >>>> correspond to a particular message ?
> >>> No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
> >>> is associated with the parameters passed to each call of request_irq().
> >>> Pass in unique private data to each call of request_irq() and the driver
> >>> can determine which message was delivered when the ISR gets called.
> >>> (ISR == Interrupt Service Routine, aka driver IRQ handler)
> >>
> >> Is it possible that even when the config space says that multiple messages
> >> are supported, you cannot enable MSI-X ?
> >
> > Yes, absolutely. Have a look at pci_msi_check_device() for starters. MSI
> > can be disabled globally, or per-device by a quirk, the bridges above
> > your device might not support MSI and the arch code may prevent
> > MSI/MSI-X for some reason (ie. platform doesn't support it).
> >
> > There's also a check in pci_enable_msix() to make sure the entries you
> > pass in are valid.
> >
> >> ------------- with MSI-X -------------
> >>
> >> saa716x_pci_init (0): found a NEMO reference board PCIe card
> >> ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
> >> PCI: Setting latency timer of device 0000:05:00.0 to 64
> >> saa716x_request_irq (0): Using MSI-X mode
> >> saa716x_enable_msix (0): MSI-X request failed
> >
> > For starters you should print the error code returned by
> > pci_enable_msix() - unfortunately it returns EINVAL for many different
> > reasons, but it will narrow it down a bit.
>
> It does return -22
If you're curious you could try this patch.
>From bd27f10c9acd24bb849d14b6fc373444322c7405 Mon Sep 17 00:00:00 2001
From: Michael Ellerman <[email protected]>
Date: Mon, 22 Dec 2008 10:21:27 +1100
Subject: [PATCH] MSI debug
---
drivers/pci/msi.c | 52 ++++++++++++++++++++++++++++++++++++++++------------
1 files changed, 40 insertions(+), 12 deletions(-)
diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 74801f7..575cca9 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -517,17 +517,26 @@ static int pci_msi_check_device(struct pci_dev* dev, int nvec, int type)
struct pci_bus *bus;
int ret;
+ if (!dev) {
+ printk(KERN_DEBUG, "MSI: null device\n");
+ return -ENODEV;
+ }
+
/* MSI must be globally enabled and supported by the device */
- if (!pci_msi_enable || !dev || dev->no_msi)
+ if (!pci_msi_enable || dev->no_msi)
+ dev_printk(KERN_DEBUG, &dev->dev, "MSI: MSI disabled\n");
return -EINVAL;
+ }
/*
* You can't ask to have 0 or less MSIs configured.
* a) it's stupid ..
* b) the list manipulation code assumes nvec >= 1.
*/
- if (nvec < 1)
+ if (nvec < 1) {
+ dev_printk(KERN_DEBUG, &dev->dev, "MSI: < 1 MSI requested\n");
return -ERANGE;
+ }
/* Any bridge which does NOT route MSI transactions from it's
* secondary bus to it's primary bus must set NO_MSI flag on
@@ -535,16 +544,24 @@ static int pci_msi_check_device(struct pci_dev* dev, int nvec, int type)
* We expect only arch-specific PCI host bus controller driver
* or quirks for specific PCI bridges to be setting NO_MSI.
*/
- for (bus = dev->bus; bus; bus = bus->parent)
- if (bus->bus_flags & PCI_BUS_FLAGS_NO_MSI)
+ for (bus = dev->bus; bus; bus = bus->parent) {
+ if (bus->bus_flags & PCI_BUS_FLAGS_NO_MSI) {
+ dev_printk(KERN_DEBUG, &dev->dev,
+ "MSI: bus check failed\n");
return -EINVAL;
+ }
+ }
ret = arch_msi_check_device(dev, nvec, type);
- if (ret)
+ if (ret) {
+ dev_printk(KERN_DEBUG, &dev->dev, "MSI: arch check failed\n");
return ret;
+ }
- if (!pci_find_capability(dev, type))
+ if (!pci_find_capability(dev, type)) {
+ dev_printk(KERN_DEBUG, &dev->dev, "MSI: no capability found\n");
return -EINVAL;
+ }
return 0;
}
@@ -669,26 +686,37 @@ int pci_enable_msix(struct pci_dev* dev, struct msix_entry *entries, int nvec)
int i, j;
u16 control;
- if (!entries)
- return -EINVAL;
-
status = pci_msi_check_device(dev, nvec, PCI_CAP_ID_MSIX);
if (status)
return status;
+ if (!entries) {
+ dev_printk(KERN_DEBUG, &dev->dev, "MSI: null entries\n");
+ return -EINVAL;
+ }
+
pos = pci_find_capability(dev, PCI_CAP_ID_MSIX);
pci_read_config_word(dev, msi_control_reg(pos), &control);
nr_entries = multi_msix_capable(control);
- if (nvec > nr_entries)
+ if (nvec > nr_entries) {
+ dev_printk(KERN_DEBUG, &dev->dev, "MSI: too many entries\n");
return -EINVAL;
+ }
/* Check for any invalid entries */
for (i = 0; i < nvec; i++) {
- if (entries[i].entry >= nr_entries)
+ if (entries[i].entry >= nr_entries) {
+ dev_printk(KERN_DEBUG, &dev->dev,
+ "MSI: invalid entry %d\n", i);
return -EINVAL; /* invalid entry */
+ }
for (j = i + 1; j < nvec; j++) {
- if (entries[i].entry == entries[j].entry)
+ if (entries[i].entry == entries[j].entry) {
+ dev_printk(KERN_DEBUG, &dev->dev,
+ "MSI: duplicate entries %d, %d\n",
+ i, j);
return -EINVAL; /* duplicate entry */
+ }
}
}
WARN_ON(!!dev->msix_enabled);
--
1.5.3.7.1.g4e596e
Michael Ellerman wrote:
> On Mon, 2008-12-22 at 02:56 +0400, Manu Abraham wrote:
>> Michael Ellerman wrote:
>>> On Sun, 2008-12-21 at 01:13 +0400, Manu Abraham wrote:
>>>> Grant Grundler wrote:
>>>>> On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
>>>>> ...
>>>>>>> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
>>>>>>> the MSI-X table. Driver then calls request_irq() to bind an interrupt
>>>>>>> handler to each GSI. So the driver never directly sees the "message".
>>>>>> Oh, you mean the array of irq_handlers in the MSI-X table should
>>>>>> correspond to a particular message ?
>>>>> No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
>>>>> is associated with the parameters passed to each call of request_irq().
>>>>> Pass in unique private data to each call of request_irq() and the driver
>>>>> can determine which message was delivered when the ISR gets called.
>>>>> (ISR == Interrupt Service Routine, aka driver IRQ handler)
>>>> Is it possible that even when the config space says that multiple messages
>>>> are supported, you cannot enable MSI-X ?
>>> Yes, absolutely. Have a look at pci_msi_check_device() for starters. MSI
>>> can be disabled globally, or per-device by a quirk, the bridges above
>>> your device might not support MSI and the arch code may prevent
>>> MSI/MSI-X for some reason (ie. platform doesn't support it).
>>>
>>> There's also a check in pci_enable_msix() to make sure the entries you
>>> pass in are valid.
>>>
>>>> ------------- with MSI-X -------------
>>>>
>>>> saa716x_pci_init (0): found a NEMO reference board PCIe card
>>>> ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
>>>> PCI: Setting latency timer of device 0000:05:00.0 to 64
>>>> saa716x_request_irq (0): Using MSI-X mode
>>>> saa716x_enable_msix (0): MSI-X request failed
>>> For starters you should print the error code returned by
>>> pci_enable_msix() - unfortunately it returns EINVAL for many different
>>> reasons, but it will narrow it down a bit.
>> It does return -22
>
> If you're curious you could try this patch.
There's a missing opening brace. I did add it and added more of kernel hacking
for debugging and loaded the modules after that, which did cause a complete
freeze of the machine.
Currently, digging into what would have caused that freeze.
The same module without the debug (kernel hack + pci debug) doesn't cause the
freeze and results in the original log that i attached some mails earlier.
(No trace/log results, results in a complete freeze immediately on module
(saa716x_hybrid) load)
Regards,
Manu
>
>>From bd27f10c9acd24bb849d14b6fc373444322c7405 Mon Sep 17 00:00:00 2001
> From: Michael Ellerman <[email protected]>
> Date: Mon, 22 Dec 2008 10:21:27 +1100
> Subject: [PATCH] MSI debug
>
> ---
> drivers/pci/msi.c | 52 ++++++++++++++++++++++++++++++++++++++++------------
> 1 files changed, 40 insertions(+), 12 deletions(-)
>
> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
> index 74801f7..575cca9 100644
> --- a/drivers/pci/msi.c
> +++ b/drivers/pci/msi.c
> @@ -517,17 +517,26 @@ static int pci_msi_check_device(struct pci_dev* dev, int nvec, int type)
> struct pci_bus *bus;
> int ret;
>
> + if (!dev) {
> + printk(KERN_DEBUG, "MSI: null device\n");
> + return -ENODEV;
> + }
> +
> /* MSI must be globally enabled and supported by the device */
> - if (!pci_msi_enable || !dev || dev->no_msi)
> + if (!pci_msi_enable || dev->no_msi)
Missing opening brace in here.
> + dev_printk(KERN_DEBUG, &dev->dev, "MSI: MSI disabled\n");
> return -EINVAL;
> + }
>
> /*
> * You can't ask to have 0 or less MSIs configured.
> * a) it's stupid ..
> * b) the list manipulation code assumes nvec >= 1.
> */
> - if (nvec < 1)
> + if (nvec < 1) {
> + dev_printk(KERN_DEBUG, &dev->dev, "MSI: < 1 MSI requested\n");
> return -ERANGE;
> + }
>
> /* Any bridge which does NOT route MSI transactions from it's
> * secondary bus to it's primary bus must set NO_MSI flag on
> @@ -535,16 +544,24 @@ static int pci_msi_check_device(struct pci_dev* dev, int nvec, int type)
> * We expect only arch-specific PCI host bus controller driver
> * or quirks for specific PCI bridges to be setting NO_MSI.
> */
> - for (bus = dev->bus; bus; bus = bus->parent)
> - if (bus->bus_flags & PCI_BUS_FLAGS_NO_MSI)
> + for (bus = dev->bus; bus; bus = bus->parent) {
> + if (bus->bus_flags & PCI_BUS_FLAGS_NO_MSI) {
> + dev_printk(KERN_DEBUG, &dev->dev,
> + "MSI: bus check failed\n");
> return -EINVAL;
> + }
> + }
>
> ret = arch_msi_check_device(dev, nvec, type);
> - if (ret)
> + if (ret) {
> + dev_printk(KERN_DEBUG, &dev->dev, "MSI: arch check failed\n");
> return ret;
> + }
>
> - if (!pci_find_capability(dev, type))
> + if (!pci_find_capability(dev, type)) {
> + dev_printk(KERN_DEBUG, &dev->dev, "MSI: no capability found\n");
> return -EINVAL;
> + }
>
> return 0;
> }
> @@ -669,26 +686,37 @@ int pci_enable_msix(struct pci_dev* dev, struct msix_entry *entries, int nvec)
> int i, j;
> u16 control;
>
> - if (!entries)
> - return -EINVAL;
> -
> status = pci_msi_check_device(dev, nvec, PCI_CAP_ID_MSIX);
> if (status)
> return status;
>
> + if (!entries) {
> + dev_printk(KERN_DEBUG, &dev->dev, "MSI: null entries\n");
> + return -EINVAL;
> + }
> +
> pos = pci_find_capability(dev, PCI_CAP_ID_MSIX);
> pci_read_config_word(dev, msi_control_reg(pos), &control);
> nr_entries = multi_msix_capable(control);
> - if (nvec > nr_entries)
> + if (nvec > nr_entries) {
> + dev_printk(KERN_DEBUG, &dev->dev, "MSI: too many entries\n");
> return -EINVAL;
> + }
>
> /* Check for any invalid entries */
> for (i = 0; i < nvec; i++) {
> - if (entries[i].entry >= nr_entries)
> + if (entries[i].entry >= nr_entries) {
> + dev_printk(KERN_DEBUG, &dev->dev,
> + "MSI: invalid entry %d\n", i);
> return -EINVAL; /* invalid entry */
> + }
> for (j = i + 1; j < nvec; j++) {
> - if (entries[i].entry == entries[j].entry)
> + if (entries[i].entry == entries[j].entry) {
> + dev_printk(KERN_DEBUG, &dev->dev,
> + "MSI: duplicate entries %d, %d\n",
> + i, j);
> return -EINVAL; /* duplicate entry */
> + }
> }
> }
> WARN_ON(!!dev->msix_enabled);
On Mon, 2008-12-22 at 19:31 +0400, Manu Abraham wrote:
> Michael Ellerman wrote:
> > On Mon, 2008-12-22 at 02:56 +0400, Manu Abraham wrote:
> >> Michael Ellerman wrote:
> >>> On Sun, 2008-12-21 at 01:13 +0400, Manu Abraham wrote:
> >>>> Grant Grundler wrote:
> >>>>> On Sun, Dec 14, 2008 at 03:15:15PM +0400, Manu Abraham wrote:
> >>>>> ...
> >>>>>>> A "GSI" (Generic Sys Interrupt?) is associated with each entry in
> >>>>>>> the MSI-X table. Driver then calls request_irq() to bind an interrupt
> >>>>>>> handler to each GSI. So the driver never directly sees the "message".
> >>>>>> Oh, you mean the array of irq_handlers in the MSI-X table should
> >>>>>> correspond to a particular message ?
> >>>>> No. I mean each MSI-X entry has an address+message pair and each MSI-X entry
> >>>>> is associated with the parameters passed to each call of request_irq().
> >>>>> Pass in unique private data to each call of request_irq() and the driver
> >>>>> can determine which message was delivered when the ISR gets called.
> >>>>> (ISR == Interrupt Service Routine, aka driver IRQ handler)
> >>>> Is it possible that even when the config space says that multiple messages
> >>>> are supported, you cannot enable MSI-X ?
> >>> Yes, absolutely. Have a look at pci_msi_check_device() for starters. MSI
> >>> can be disabled globally, or per-device by a quirk, the bridges above
> >>> your device might not support MSI and the arch code may prevent
> >>> MSI/MSI-X for some reason (ie. platform doesn't support it).
> >>>
> >>> There's also a check in pci_enable_msix() to make sure the entries you
> >>> pass in are valid.
> >>>
> >>>> ------------- with MSI-X -------------
> >>>>
> >>>> saa716x_pci_init (0): found a NEMO reference board PCIe card
> >>>> ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 19
> >>>> PCI: Setting latency timer of device 0000:05:00.0 to 64
> >>>> saa716x_request_irq (0): Using MSI-X mode
> >>>> saa716x_enable_msix (0): MSI-X request failed
> >>> For starters you should print the error code returned by
> >>> pci_enable_msix() - unfortunately it returns EINVAL for many different
> >>> reasons, but it will narrow it down a bit.
> >> It does return -22
> >
> > If you're curious you could try this patch.
>
>
> There's a missing opening brace. I did add it and added more of kernel hacking
> for debugging and loaded the modules after that, which did cause a complete
> freeze of the machine.
Yeah sorry I didn't compile it. I don't see how it would have caused the
kernel to hang though.
cheers
--
Michael Ellerman
OzLabs, IBM Australia Development Lab
wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)
We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person
On Mon, Dec 15, 2008 at 04:07:53PM +0900, Kenji Kaneshige wrote:
> Grant Grundler wrote:
> >> To put it short: i am wondering how i should read the MSI messages.
> >
> > The "message" is actually mapped to an the interrupt vector by the core
> > generic interrupt handling code in the kernel.
> >
> > A "GSI" (Generic Sys Interrupt?) is associated with each entry in
> > the MSI-X table. Driver then calls request_irq() to bind an interrupt
> > handler to each GSI. So the driver never directly sees the "message".
> >
>
> I think "GSI (Global System Interrupt)" is for identifying the I/O
> APIC pin among multiple I/O APICs. Maybe you wanted to mean the
> interrupt number managed by kernel (frequently called "IRQ")?
I thought "GSI" and "kernel interrupt number" are equivalents.
But I could easily be confusing the terms here. If "GSI" is only
used for identifying APIC input lines, then that's not right term.
> Or am I misunderstanding something?
You probably understand it better than I do. :)
thanks,
grant