Do you have a device that doesn't work unless you specify
"pci=routeirq"?
If all your devices work, you can ignore the note in dmesg.
On Tue, 4 Jan 2005, Bjorn Helgaas wrote:
> Do you have a device that doesn't work unless you specify
> "pci=routeirq"?
>
> If all your devices work, you can ignore the note in dmesg.
>
> -
I note that pci_enable_device() needs to be executed __before__
the IRQ is obtained on 2.6.10, otherwise you get the wrong IRQ
(IRQ10 on this system)B.
This doesn't seem to be correct since the IRQ connection was set
by the BIOS and certainly shouldn't be changed. On this system,
interrupts that were not shared on 2.4.n and early 2.6.n end
up being shared... See IRQ18 below.
CPU0
0: 1135520 IO-APIC-edge timer
1: 3203 IO-APIC-edge i8042
7: 0 IO-APIC-edge parport0
8: 1 IO-APIC-edge rtc
9: 0 IO-APIC-level acpi
12: 66 IO-APIC-edge i8042
14: 10637 IO-APIC-edge ide0
16: 0 IO-APIC-level uhci_hcd, uhci_hcd
18: 81 IO-APIC-level libata, uhci_hcd, Analogic Corp DLB
19: 0 IO-APIC-level uhci_hcd
20: 801 IO-APIC-level eth0
21: 982 IO-APIC-level aic7xxx
23: 0 IO-APIC-level ehci_hcd
NMI: 0
LOC: 1135484
ERR: 0
MIS: 0
Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.
On Tue, 2005-01-04 at 13:53 -0500, linux-os wrote:
> I note that pci_enable_device() needs to be executed __before__
> the IRQ is obtained on 2.6.10, otherwise you get the wrong IRQ
> (IRQ10 on this system)B.
Right.
> This doesn't seem to be correct since the IRQ connection was set
> by the BIOS and certainly shouldn't be changed. On this system,
> interrupts that were not shared on 2.4.n and early 2.6.n end
> up being shared... See IRQ18 below.
It's not that we are changing the IRQ, it's just that we now
do the ACPI routing at the time the driver claims the device,
rather than doing all the ACPI routing at boot-time. The
old strategy messed with IRQs that might never be used (which
broke some things), and also didn't work for hot-plug PCI
root bridges.
Back to my original question, do you have a device that
only works when you use "pci=routeirq"? If so, what is
it and what driver does it use?
On Tue, 4 Jan 2005, Bjorn Helgaas wrote:
> On Tue, 2005-01-04 at 13:53 -0500, linux-os wrote:
>
>> I note that pci_enable_device() needs to be executed __before__
>> the IRQ is obtained on 2.6.10, otherwise you get the wrong IRQ
>> (IRQ10 on this system)B.
>
> Right.
>
>> This doesn't seem to be correct since the IRQ connection was set
>> by the BIOS and certainly shouldn't be changed. On this system,
>> interrupts that were not shared on 2.4.n and early 2.6.n end
>> up being shared... See IRQ18 below.
>
> It's not that we are changing the IRQ, it's just that we now
> do the ACPI routing at the time the driver claims the device,
> rather than doing all the ACPI routing at boot-time. The
> old strategy messed with IRQs that might never be used (which
> broke some things), and also didn't work for hot-plug PCI
> root bridges.
>
> Back to my original question, do you have a device that
> only works when you use "pci=routeirq"? If so, what is
> it and what driver does it use?
>
No.
I modified our drivers to accommodate the new scheme. The problem
is that I don't feel warm and fuzzy about enabling a device
__before__ an IRQ handler is in place to handle the IRQ.
For instance, Level interrupts from PLX chips on the PCI bus
can (read do) generate interrupts when some of the BARS are
being configured. Once you get an unhandled interrupt, you
are dead because there's nothing to reset the line.
The new scheme requires that the device be enabled to get
the correct IRQ. If you did this work, maybe it would
be much better if you added a pci_set_irq() call instead
of combining your routing with enabling the device?
Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.
linux-os wrote:
>
> For instance, Level interrupts from PLX chips on the PCI bus
> can (read do) generate interrupts when some of the BARS are
> being configured. Once you get an unhandled interrupt, you
> are dead because there's nothing to reset the line.
Why not unconditionally clear all interrupts after configuring the chip?
David Vrabel
On Wed, 5 Jan 2005, David Vrabel wrote:
> linux-os wrote:
>>
>> For instance, Level interrupts from PLX chips on the PCI bus
>> can (read do) generate interrupts when some of the BARS are
>> being configured. Once you get an unhandled interrupt, you
>> are dead because there's nothing to reset the line.
>
> Why not unconditionally clear all interrupts after configuring the chip?
>
> David Vrabel
>
You can't configure the chip until the BARS are set up for
access. You _must_ know what the interrupt line is, before
you touch any registers so that any "waiting" interrupt
gets handled, i.e., cleared. Otherwise, the code that
inspects the PCI bus, looking for a device to claim, finds
the device, then (in order to make its IRQ correct) enables
it ... BAM that's all she wrote. No messages, no nothing,
a halted machine because the IRQ line is permanently TRUE
with no code in place to reset it.
The temporary work-around is....
pci_enable_device(pdev);
save_irq = pdev->irq;
pci_disable_device(pdev); // Turn back off.
init_bars(....);
request_irq(save_irq,...) // Put ISR in place
pci_write_config_byte(pdev, PCI_CACHE_LINE_SIZE, 0x08);
pci_write_config_byte(pdev, PCI_LATENCY_TIMER, 0x40);
pci_set_dma_mask(pdev, 0x00000000ffffffffULL);
pci_set_drvdata(pdev, NULL);
pci_set_power_state(pdev, 0);
pci_set_master(pdev);
pci_set_mwi(pdev);
pci_write_config_dword(pdev, PCI_COMMAND, PCI_CONFIG);
.... configure chip-specific stuff, clear interrupts, etc.
pci_enable_device(dev);
Now, the temporary work-around is a MACRO called ROUTE_IRQ().
If the guy who changed the PCI API adds a seperate callable
procedure to do this routing without actually enabling the
chip, I will substitute.
This work-round is a new Linux-2.6.10 bug-hack. It was never
required before
Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.
On Wed, 2005-01-05 at 07:06 -0500, linux-os wrote:
> The temporary work-around is....
> pci_enable_device(pdev);
> save_irq = pdev->irq;
> pci_disable_device(pdev); // Turn back off.
>
> init_bars(....);
> request_irq(save_irq,...) // Put ISR in place
>
> pci_write_config_byte(pdev, PCI_CACHE_LINE_SIZE, 0x08);
> pci_write_config_byte(pdev, PCI_LATENCY_TIMER, 0x40);
> pci_set_dma_mask(pdev, 0x00000000ffffffffULL);
> pci_set_drvdata(pdev, NULL);
> pci_set_power_state(pdev, 0);
> pci_set_master(pdev);
> pci_set_mwi(pdev);
> pci_write_config_dword(pdev, PCI_COMMAND, PCI_CONFIG);
>
> .... configure chip-specific stuff, clear interrupts, etc.
> pci_enable_device(dev);
So prior to 2.6.10, you did something like this?
request_irq(pdev->irq, ...);
pci_write_config_byte(pdev, PCI_CACHE_LINE_SIZE, 0x08);
pci_write_config_byte(pdev, PCI_LATENCY_TIMER, 0x40);
...
pci_enable_device(pdev);
What sort of interrupts does the device generate before it's
enabled? I can't find anything in the PCI spec that actually
prohibits interrupts before the driver starts up the device,
but it does seem strange.
You wouldn't want your ISR mucking around with a half-initialized
device, so does it have to check a "device_configured" flag
or something?
On Wed, 5 Jan 2005, Bjorn Helgaas wrote:
> On Wed, 2005-01-05 at 07:06 -0500, linux-os wrote:
>> The temporary work-around is....
>> pci_enable_device(pdev);
>> save_irq = pdev->irq;
>> pci_disable_device(pdev); // Turn back off.
>>
>> init_bars(....);
>> request_irq(save_irq,...) // Put ISR in place
>>
>> pci_write_config_byte(pdev, PCI_CACHE_LINE_SIZE, 0x08);
>> pci_write_config_byte(pdev, PCI_LATENCY_TIMER, 0x40);
>> pci_set_dma_mask(pdev, 0x00000000ffffffffULL);
>> pci_set_drvdata(pdev, NULL);
>> pci_set_power_state(pdev, 0);
>> pci_set_master(pdev);
>> pci_set_mwi(pdev);
>> pci_write_config_dword(pdev, PCI_COMMAND, PCI_CONFIG);
>>
>> .... configure chip-specific stuff, clear interrupts, etc.
>> pci_enable_device(dev);
>
> So prior to 2.6.10, you did something like this?
>
> request_irq(pdev->irq, ...);
> pci_write_config_byte(pdev, PCI_CACHE_LINE_SIZE, 0x08);
> pci_write_config_byte(pdev, PCI_LATENCY_TIMER, 0x40);
> ...
> pci_enable_device(pdev);
>
Yes, exactly.
> What sort of interrupts does the device generate before it's
> enabled? I can't find anything in the PCI spec that actually
> prohibits interrupts before the driver starts up the device,
> but it does seem strange.
>
The problem is that the PLX-9656BA INTCSR is not in configuration
space, but runtime registers off a BAR. The interrupt source
can be from a PLD that hasn't even had its microcode loaded
yet!
FYI, the PLX or similar clone is the bus interface chip for many
busmastering PCI boards.
> You wouldn't want your ISR mucking around with a half-initialized
> device, so does it have to check a "device_configured" flag
> or something?
>
Yes. If the device isn't configured, the ISR reads all the INTCSR
bits, then writes 0 to the register to prevent anything else.
If the PLX had been reset, then the INTCSR bits would all
be masked off. However, reset is really only guaranteed from
power OFF on some motherboards, in particuar the ones with
so-called "hot-swap" capabilites fail. There is a software
reset that, in fact, even reloads its serial EEPROM. However,
the BAR needs to be accessible for this to be used.
So it would be wonderful if the correct IRQ could be made
available before the chip could generate an interrupt.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.
On Wed, 2005-01-05 at 13:15 -0500, linux-os wrote:
> The problem is that the PLX-9656BA INTCSR is not in configuration
> space, but runtime registers off a BAR. The interrupt source
> can be from a PLD that hasn't even had its microcode loaded
> yet!
>
> FYI, the PLX or similar clone is the bus interface chip for many
> busmastering PCI boards.
>
> > You wouldn't want your ISR mucking around with a half-initialized
> > device, so does it have to check a "device_configured" flag
> > or something?
>
> Yes. If the device isn't configured, the ISR reads all the INTCSR
> bits, then writes 0 to the register to prevent anything else.
The PLX might be a common device, but it sounds like this
particular issue depends on the design of the rest of the
board. And presumably, nobody who cared about performance
would design a board with this property, right? I mean, to
add a test in the ISR for a condition that exists only for
a few milliseconds at driver startup-time seems sub-optimal.
> If the PLX had been reset, then the INTCSR bits would all
> be masked off. However, reset is really only guaranteed from
> power OFF on some motherboards, in particuar the ones with
> so-called "hot-swap" capabilites fail. There is a software
> reset that, in fact, even reloads its serial EEPROM. However,
> the BAR needs to be accessible for this to be used.
>
> So it would be wonderful if the correct IRQ could be made
> available before the chip could generate an interrupt.
If we exposed a new pcibios_route_irq() (to hide the arch-
specific nature of IRQ routing via ACPI or other information),
could you do what you need in a pci_fixup_early quirk?
On Mon, 10 Jan 2005, Bjorn Helgaas wrote:
> On Wed, 2005-01-05 at 13:15 -0500, linux-os wrote:
>> The problem is that the PLX-9656BA INTCSR is not in configuration
>> space, but runtime registers off a BAR. The interrupt source
>> can be from a PLD that hasn't even had its microcode loaded
>> yet!
>>
>> FYI, the PLX or similar clone is the bus interface chip for many
>> busmastering PCI boards.
>>
>>> You wouldn't want your ISR mucking around with a half-initialized
>>> device, so does it have to check a "device_configured" flag
>>> or something?
>>
>> Yes. If the device isn't configured, the ISR reads all the INTCSR
>> bits, then writes 0 to the register to prevent anything else.
>
> The PLX might be a common device, but it sounds like this
> particular issue depends on the design of the rest of the
> board. And presumably, nobody who cared about performance
> would design a board with this property, right? I mean, to
> add a test in the ISR for a condition that exists only for
> a few milliseconds at driver startup-time seems sub-optimal.
>
I'm not so sure about that. There are many hardware registers
to be read in the ISR before the ISR can "decide" what it is
supposed to do. I'm not sure that the few hundred nanoseconds
required to read/test a variable is all the consequential.
What I do know is that the board must recover from all known
errors because you can't irradiate a patient with X-Rays and
then decide to throw away the data.
Also, like most bus-mastering devices, you only get an interrupt
to report something, basically one interrupt per transmitted
packet and one per received. There are a few diagnostic thing
that make more, but they are not common events. The 'packets'
are 10 megabytes in length so the interrupts are really hidden
in the noise. Even Ethernet drivers with their 1500 max byte
packets that use this interface chip should really check for
a "real" interrupt now that hot-swap is commonplace.
>> If the PLX had been reset, then the INTCSR bits would all
>> be masked off. However, reset is really only guaranteed from
>> power OFF on some motherboards, in particuar the ones with
>> so-called "hot-swap" capabilites fail. There is a software
>> reset that, in fact, even reloads its serial EEPROM. However,
>> the BAR needs to be accessible for this to be used.
>>
>> So it would be wonderful if the correct IRQ could be made
>> available before the chip could generate an interrupt.
>
> If we exposed a new pcibios_route_irq() (to hide the arch-
> specific nature of IRQ routing via ACPI or other information),
> could you do what you need in a pci_fixup_early quirk?
>
That would be wonderful. If you have a patch I will install
it immediately and change the content of my macro (the source-code
even compiles on 2.4.x so there are lots of macros).
Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.