Hello,
On Linux when interrupts are defined similar to below, what defines say
ide2, ide3 to be on the same interrupt? The bios, linux, the driver using
the interrupt? And can that be controlled/overrode at the
kernel/driver level?
I have identified that the disks that are shared on ide2, ide3 do funny
things when both are being heavily used (dma_expiry), this is an older
driver versions
but I have experienced it before with a lot newer driver, and a bios
adjustment
previously fixed a similar issue, so that may be what is needed in this
case also,
I am not sure how they fixed it, but I suspect that the setup the interrupt
to not be shared. I have a large number of machines and under heavy
loads all
seem to duplicate the issue, and it always happens with the disks on
ide2/ide3,
never on the disk connected to ide4.
CPU0 CPU1 CPU2 CPU3
0: 56616921 5359998 7002142 938817 XT-PIC timer
1: 8 88 96 0 IO-APIC-edge i8042
2: 0 0 0 0 XT-PIC cascade
4: 2091 100 208 2477 IO-APIC-edge serial
8: 0 0 0 0 IO-APIC-edge rtc
9: 0 0 0 0 IO-APIC-level acpi
20: 0 0 0 0 IO-APIC-level ehci_hcd
21: 0 950 401419 414482 IO-APIC-level ide4,
ohci_hcd
22: 1165 1704243 576247 6796 IO-APIC-level ide2,
ide3
47: 65971 0 0 0 IO-APIC-level eth0
NMI: 1 1 1 1
LOC: 69904264 69877733 69879541 69901903
ERR: 0
MIS: 105
Roger
Hello,
Roger Heflin wrote:
> On Linux when interrupts are defined similar to below, what defines say
> ide2, ide3 to be on the same interrupt? The bios, linux, the driver
> using the interrupt?
In this particular case (I suppoer both belong to the same IDE conrtoller)
the hardware design determines this in part and software in another part.
If the PCI IDE controller was in the leagcy mode (it should be ide0/1 then),
the interrupts would've been 14 and 15. But in the native PCI mode, both
chanels share the same interrupt level (if they're both in native mode that is).
> And can that be controlled/overrode at the kernel/driver level?
The only possibility might be do disable ide0/1 in the BIOS I guess.
> I have identified that the disks that are shared on ide2, ide3 do funny
> things when both are being heavily used (dma_expiry), this is an older
> driver versions
> but I have experienced it before with a lot newer driver, and a bios
> adjustment
> previously fixed a similar issue, so that may be what is needed in this
> case also,
> I am not sure how they fixed it, but I suspect that the setup the interrupt
> to not be shared.
I doubt that your suspicions are justified.
> I have a large number of machines and under heavy
> loads all
> seem to duplicate the issue, and it always happens with the disks on
> ide2/ide3,
> never on the disk connected to ide4.
BTW, you never named your particular IDE hardware.
> CPU0 CPU1 CPU2 CPU3
> 0: 56616921 5359998 7002142 938817 XT-PIC timer
> 1: 8 88 96 0 IO-APIC-edge i8042
> 2: 0 0 0 0 XT-PIC cascade
> 4: 2091 100 208 2477 IO-APIC-edge serial
> 8: 0 0 0 0 IO-APIC-edge rtc
> 9: 0 0 0 0 IO-APIC-level acpi
> 20: 0 0 0 0 IO-APIC-level ehci_hcd
> 21: 0 950 401419 414482 IO-APIC-level ide4,
> ohci_hcd
> 22: 1165 1704243 576247 6796 IO-APIC-level ide2,
> ide3
> 47: 65971 0 0 0 IO-APIC-level eth0
> NMI: 1 1 1 1
> LOC: 69904264 69877733 69879541 69901903
> ERR: 0
> MIS: 105
MBR, Sergei
Roger Heflin wrote:
> Hello,
>
> On Linux when interrupts are defined similar to below, what defines say
> ide2, ide3 to be on the same interrupt? The bios, linux, the driver
> using
> the interrupt? And can that be controlled/overrode at the
> kernel/driver level?
>
Typically this is determined by the hardware routing of the interrupt
lines on the motherboard, occasionally the BIOS. There isn't usually
much that can be done about the sharing at the kernel level.
--
Robert Hancock Saskatoon, SK, Canada
To email, remove "nospam" from [email protected]
Home Page: http://www.roberthancock.com/
Sergei Shtylyov wrote:
> Hello,
>
>> I have identified that the disks that are shared on ide2, ide3 do funny
>> things when both are being heavily used (dma_expiry), this is an older
>> driver versions
>> but I have experienced it before with a lot newer driver, and a bios
>> adjustment
>> previously fixed a similar issue, so that may be what is needed in
>> this case also,
>> I am not sure how they fixed it, but I suspect that the setup the
>> interrupt
>> to not be shared.
>
> I doubt that your suspicions are justified.
>
>> I have a large number of machines and under heavy loads all
>> seem to duplicate the issue, and it always happens with the disks on
>> ide2/ide3,
>> never on the disk connected to ide4.
>
> BTW, you never named your particular IDE hardware.
NVIDIA, CK804 chipset.
And the disks are on the sata part, and are found as e g and i so if
they are in legacy mode, it would make me suspect that the disks believe
they are on separate channels, and the performance characteristics appear
to confirm that as the disks don't seem to interfere with each other.
>
>> CPU0 CPU1 CPU2 CPU3
>> 0: 56616921 5359998 7002142 938817 XT-PIC timer
>> 1: 8 88 96 0 IO-APIC-edge i8042
>> 2: 0 0 0 0 XT-PIC cascade
>> 4: 2091 100 208 2477 IO-APIC-edge serial
>> 8: 0 0 0 0 IO-APIC-edge rtc
>> 9: 0 0 0 0 IO-APIC-level acpi
>> 20: 0 0 0 0 IO-APIC-level
>> ehci_hcd
>> 21: 0 950 401419 414482 IO-APIC-level
>> ide4, ohci_hcd
>> 22: 1165 1704243 576247 6796 IO-APIC-level
>> ide2, ide3
>> 47: 65971 0 0 0 IO-APIC-level eth0
>> NMI: 1 1 1 1
>> LOC: 69904264 69877733 69879541 69901903
>> ERR: 0
>> MIS: 105
>
> MBR, Sergei
>
Ar Maw, 2006-08-15 am 09:17 -0500, ysgrifennodd Roger Heflin:
> On Linux when interrupts are defined similar to below, what defines say
> ide2, ide3 to be on the same interrupt? The bios, linux, the driver using
> the interrupt? And can that be controlled/overrode at the
> kernel/driver level?
Only with a soldering iron. They are the way the system is wired. Moving
boards between slots may change the IRQ allocation.
> I have identified that the disks that are shared on ide2, ide3 do funny
> things when both are being heavily used (dma_expiry), this is an older
> driver versions
That could be occuring just through lack of PCI bus bandwidth.
On Tuesday 15 August 2006 10:17, Roger Heflin wrote:
> 22: ? ? ? 1165 ? ?1704243 ? ? 576247 ? ? ? 6796 ? IO-APIC-level ?ide2, ide3
The first thing that determines interrupt sharing is where the physical wires from
the devices are routed. If these ide controllers are add-on boards, then you
could try moving them between PCI slots -- as the slots generally have different
primary interrupt lines. When the board runs out of unique lines,
they are generally wired to re-use lines on different slots.
The manual for the board will generally tell you how the lines
are routed.
If the devices are using the same physical interrupt line, then it
is not possible for software or BIOS to move them to different
lines.
If these devices are functions in the same device, it is possible
that there is some internal (BIOS or firmware) setting to tell it to
use two interrupt wires instead of one.
Once the wire for the device is determined, it is up to the BIOS
and the OS to program (or not) an interrupt router to assign
the wire to an interrupt input pin. Your dmesg will tell us
if this is happening on this board.
-Len
On Tue, 15 Aug 2006, Roger Heflin wrote:
> Hello,
>
> On Linux when interrupts are defined similar to below, what defines say
> ide2, ide3 to be on the same interrupt? The bios, linux, the driver using
> the interrupt? And can that be controlled/overrode at the
> kernel/driver level?
>
> I have identified that the disks that are shared on ide2, ide3 do funny
> things when both are being heavily used (dma_expiry), this is an older
> driver versions
> but I have experienced it before with a lot newer driver, and a bios
> adjustment
> previously fixed a similar issue, so that may be what is needed in this
> case also,
> I am not sure how they fixed it, but I suspect that the setup the interrupt
> to not be shared. I have a large number of machines and under heavy
> loads all
> seem to duplicate the issue, and it always happens with the disks on
> ide2/ide3,
> never on the disk connected to ide4.
>
> CPU0 CPU1 CPU2 CPU3
> 0: 56616921 5359998 7002142 938817 XT-PIC timer
> 1: 8 88 96 0 IO-APIC-edge i8042
> 2: 0 0 0 0 XT-PIC cascade
> 4: 2091 100 208 2477 IO-APIC-edge serial
> 8: 0 0 0 0 IO-APIC-edge rtc
> 9: 0 0 0 0 IO-APIC-level acpi
> 20: 0 0 0 0 IO-APIC-level ehci_hcd
> 21: 0 950 401419 414482 IO-APIC-level ide4,
> ohci_hcd
> 22: 1165 1704243 576247 6796 IO-APIC-level ide2,
> ide3
> 47: 65971 0 0 0 IO-APIC-level eth0
> NMI: 1 1 1 1
> LOC: 69904264 69877733 69879541 69901903
> ERR: 0
> MIS: 105
>
>
> Roger
The hardware defines what traces go to whatever chips on the
motherboard and to the feature module sockets. Sometimes,
the ultimate IRQ that these traces go to can be configured in the
BIOS. Sometimes not. The kernel just uses what was set up by
the BIOS, convention, and the PCI devices themselves.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.24 on an i686 machine (5592.62 BogoMips).
New book: http://www.AbominableFirebug.com/
_
****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Thank you.
Alan Cox wrote:
> Ar Maw, 2006-08-15 am 09:17 -0500, ysgrifennodd Roger Heflin:
>> On Linux when interrupts are defined similar to below, what defines say
>> ide2, ide3 to be on the same interrupt? The bios, linux, the driver using
>> the interrupt? And can that be controlled/overrode at the
>> kernel/driver level?
>
> Only with a soldering iron. They are the way the system is wired. Moving
> boards between slots may change the IRQ allocation.
>
>> I have identified that the disks that are shared on ide2, ide3 do funny
>> things when both are being heavily used (dma_expiry), this is an older
>> driver versions
>
> That could be occuring just through lack of PCI bus bandwidth.
>
>
>
Right now, all controllers are part of the NVIDIA chipset which should
mean the are on the same PCI bandwidth. The block diagram does indicate
the 4 sata channels as one unit, though I don't know exactly how things
were internally done, performance testing seems to indicate that all
3 I am using are independent hardware as they don't seem to affect each
others load.
I am currently retesting under 2.6.17.8 to see if I have similar issues
there, under that it show interrupts like below, different interrupt
numbers,
but similar sharing as ata1/ata2, and ata3/ata4 are on the same interrupt.
ata1: SATA max UDMA/133 cmd 0xEC00 ctl 0xE802 bmdma 0xDC00 irq 16
ata2: SATA max UDMA/133 cmd 0xE400 ctl 0xE002 bmdma 0xDC08 irq 16
ata1: SATA link up 3.0 Gbps (SStatus 123)
ata1: dev 0 cfg 49:2f00 82:706b 83:7e61 84:4023 85:7069 86:3e41 87:4023
88:407f
ata1: dev 0 ATA-7, max UDMA/133, 321672960 sectors: LBA48
ata1: dev 0 configured for UDMA/133
ata2: SATA link up 3.0 Gbps (SStatus 123)
ata2: dev 0 cfg 49:2f00 82:706b 83:7e61 84:4023 85:7069 86:3e41 87:4023
88:407f
ata2: dev 0 ATA-7, max UDMA/133, 321672960 sectors: LBA48
ata2: dev 0 configured for UDMA/133
ata3: SATA max UDMA/133 cmd 0xD800 ctl 0xD402 bmdma 0xC800 irq 17
ata4: SATA max UDMA/133 cmd 0xD000 ctl 0xCC02 bmdma 0xC808 irq 17
ata3: SATA link up 3.0 Gbps (SStatus 123)
ata3: dev 0 cfg 49:2f00 82:706b 83:7e61 84:4023 85:7069 86:3e41 87:4023
88:407f
ata3: dev 0 ATA-7, max UDMA/133, 321672960 sectors: LBA48
ata3: dev 0 configured for UDMA/133
ata4: SATA link down (SStatus 0)
Roger
Ar Maw, 2006-08-15 am 10:24 -0500, ysgrifennodd Roger Heflin:
> I am currently retesting under 2.6.17.8 to see if I have similar issues
> there, under that it show interrupts like below, different interrupt
> numbers,
> but similar sharing as ata1/ata2, and ata3/ata4 are on the same interrupt.
Thats what I would expect to see - two channels per PCI device is the
normal layout and they will always share the IRQ.
we've seen a lot of problems on ck804 chipsets when multiple devices
share level-triggered interrupts. I think some of the earlier sample
bioses assumed that interrupts would be configured via ACPI, and when
ACPI is not used, the interrupts end up as level-triggered instead of
edge-triggered.
you might investigate whether ACPI is failing to configure your system
(are you on 2.4? updating to 2.6 might help), using the 'noapic'
kernel parameter to force interrupts back to PIC, or the 'irqpoll' kernel
parameter to try to work around the level-triggered interrupts.
Thanks,
Terence
On Tue, Aug 15, 2006 at 09:17:04AM -0500, [email protected] wrote:
> Hello,
>
> On Linux when interrupts are defined similar to below, what defines say
> ide2, ide3 to be on the same interrupt? The bios, linux, the driver using
> the interrupt? And can that be controlled/overrode at the
> kernel/driver level?
>
> I have identified that the disks that are shared on ide2, ide3 do funny
> things when both are being heavily used (dma_expiry), this is an older
> driver versions
> but I have experienced it before with a lot newer driver, and a bios
> adjustment
> previously fixed a similar issue, so that may be what is needed in this
> case also,
> I am not sure how they fixed it, but I suspect that the setup the interrupt
> to not be shared. I have a large number of machines and under heavy
> loads all
> seem to duplicate the issue, and it always happens with the disks on
> ide2/ide3,
> never on the disk connected to ide4.
>
> CPU0 CPU1 CPU2 CPU3
> 0: 56616921 5359998 7002142 938817 XT-PIC timer
> 1: 8 88 96 0 IO-APIC-edge i8042
> 2: 0 0 0 0 XT-PIC cascade
> 4: 2091 100 208 2477 IO-APIC-edge serial
> 8: 0 0 0 0 IO-APIC-edge rtc
> 9: 0 0 0 0 IO-APIC-level acpi
> 20: 0 0 0 0 IO-APIC-level ehci_hcd
> 21: 0 950 401419 414482 IO-APIC-level ide4,
> ohci_hcd
> 22: 1165 1704243 576247 6796 IO-APIC-level ide2,
> ide3
> 47: 65971 0 0 0 IO-APIC-level eth0
> NMI: 1 1 1 1
> LOC: 69904264 69877733 69879541 69901903
> ERR: 0
> MIS: 105
>
>
> Roger
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
>
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------
Hello.
Terence Ripperda wrote:
> we've seen a lot of problems on ck804 chipsets when multiple devices
> share level-triggered interrupts. I think some of the earlier sample
> bioses assumed that interrupts would be configured via ACPI, and when
> ACPI is not used, the interrupts end up as level-triggered instead of
> edge-triggered.
Edge-triggered *shared* interrupts?! Now that sounds interesting (I'm not
saying impossible).
WBR, Sergei
On Tue, Aug 15, 2006 at 09:42:49PM +0400, [email protected] wrote:
> Hello.
>
> Terence Ripperda wrote:
> >we've seen a lot of problems on ck804 chipsets when multiple devices
> >share level-triggered interrupts. I think some of the earlier sample
> >bioses assumed that interrupts would be configured via ACPI, and when
> >ACPI is not used, the interrupts end up as level-triggered instead of
> >edge-triggered.
>
> Edge-triggered *shared* interrupts?! Now that sounds interesting (I'm
> not saying impossible).
ah shoot, you're right. doing too much and got the two backwards.
ignore my email, I'm going off to sit in the corner with a dunce cap
on..
>
> WBR, Sergei
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------
Alan Cox wrote:
> Ar Maw, 2006-08-15 am 10:24 -0500, ysgrifennodd Roger Heflin:
>
>> I am currently retesting under 2.6.17.8 to see if I have similar issues
>> there, under that it show interrupts like below, different interrupt
>> numbers,
>> but similar sharing as ata1/ata2, and ata3/ata4 are on the same interrupt.
>
> Thats what I would expect to see - two channels per PCI device is the
> normal layout and they will always share the IRQ.
>
>
After looking at other information that looks correct.
It looks like the older DMA recovery code never works on this chipset,
once it goes into DMA recovery it never comes out of it. I am looking
at that to see if anything can be done about it.
The problem I am having is that we cannot use a later kernel because it
has some other issues (MPT fusion driver is 1/4 the speed of the older
one for some unknown reason-this is in 2.6.17, the older MPT fusion works
fine on exactly the same machine/cards/bios-I am not sure of the
actual underlying casue of this slowdown).
On the specific kernel that I have I appear to have both IDE and
sata_nv drivers, is there a way to force things to use sata_nv/libata
rather than the older ide driver for the NVIDIA sata controller?
Roger
Ar Maw, 2006-08-15 am 12:47 -0500, ysgrifennodd Roger Heflin:
> On the specific kernel that I have I appear to have both IDE and
> sata_nv drivers, is there a way to force things to use sata_nv/libata
> rather than the older ide driver for the NVIDIA sata controller?
You can build 2.6.18-mm like that (say N to AMD/NVIDIA PATA driver, y to
sata_nv and pata_amd libata drivers). Still have to figure out why your
MPT fusion is slow but if its still slow in 2.6.18-mm then it needs
figuring out anyway
Roger Heflin wrote:
> On the specific kernel that I have I appear to have both IDE and
> sata_nv drivers, is there a way to force things to use sata_nv/libata
> rather than the older ide driver for the NVIDIA sata controller?
Are you saying that drivers/ide is binding to the NVIDIA SATA
controller? That seems odd, at least in the Fedora Core 5 kernel
configuration, drivers/ide would never try to bind to the SATA
controllers for me, only sata_nv would. Maybe you have some
configuration option turned on to make drivers/ide grab anything that
looks like an IDE controller, which probably shouldn't be turned on.
--
Robert Hancock Saskatoon, SK, Canada
To email, remove "nospam" from [email protected]
Home Page: http://www.roberthancock.com/
Robert Hancock wrote:
> Roger Heflin wrote:
>> On the specific kernel that I have I appear to have both IDE and
>> sata_nv drivers, is there a way to force things to use sata_nv/libata
>> rather than the older ide driver for the NVIDIA sata controller?
>
> Are you saying that drivers/ide is binding to the NVIDIA SATA
> controller? That seems odd, at least in the Fedora Core 5 kernel
> configuration, drivers/ide would never try to bind to the SATA
> controllers for me, only sata_nv would. Maybe you have some
> configuration option turned on to make drivers/ide grab anything that
> looks like an IDE controller, which probably shouldn't be turned on.
>
This kernel is a bit weird, ide is build into the kernel so it gets
first shot at anything it can manage. I am not sure why the choices
were made to set it up the way it is setup, probably something historical.
I adjusted it so that IDE is a module and setup IDE to load after
sata_nv, so sata_nv is picking up the NVIDIA stuff, and it appears to
be alot better behaved.
Roger
Roger Heflin wrote:
>
> It looks like the older DMA recovery code never works on this chipset,
> once it goes into DMA recovery it never comes out of it. I am looking
> at that to see if anything can be done about it.
You could try booting with ide2=serialize as a kernel parameter.
That should ensure the two channels are never in use simultaneously.
Trades off a bit of performance for a bit of reliability.
Cheers
Ar Mer, 2006-08-16 am 18:54 -0400, ysgrifennodd Mark Lord:
> Roger Heflin wrote:
> >
> > It looks like the older DMA recovery code never works on this chipset,
> > once it goes into DMA recovery it never comes out of it.
DMA recovery is fairly broken in drivers/ide especially if it tries to
change mode. libata does not have this problem and I have no plans to
even try and fix the drivers/ide code for this issue as its a major
piece of work.
Alan Cox wrote:
> Ar Mer, 2006-08-16 am 18:54 -0400, ysgrifennodd Mark Lord:
>> Roger Heflin wrote:
>>> It looks like the older DMA recovery code never works on this chipset,
>>> once it goes into DMA recovery it never comes out of it.
>
> DMA recovery is fairly broken in drivers/ide especially if it tries to
> change mode. libata does not have this problem and I have no plans to
> even try and fix the drivers/ide code for this issue as its a major
> piece of work.
>
>
If already figured that since I was pretty sure all of it was
being rolled to libata. I was able to switch that controller and
disk to the libata sata_nv.
Roger