Hello all,
We have a proprietary PCI board installed in a (UP) system with an ASUS P4PE
motherboard (uses Intel 845PE chipset). This system is running Red Hat 9.0
(kernel version 2.4.20-8). I've combed the ASUS website, the linux-kernel
mailing lists and the internet in general for answers, but have turned up
nothing.
Here's what I've found: I've discovered that our software will run for a
short time and then appear to be hung (keyboard and mouse not responding,
can't remote log in). We have found that when it hangs, our software is
still
running and there are no reports of Linux Oops, system panics or processor
faults. At that time, our driver is expecting an interrupt indicating a DMA
completion, but times out because it never receives it. It then disables
the interrupt at the board and issues a reset of the board's interrupt.
Then it tries sending it's data again.
At the point of failure, the logic analyzer shows that our board is
asserting the interrupt. It also shows that on all _previous_ interrupts
(and in the case of our timeout), I clear the interrupt appropriatly on our
board. In an effort to find out why the driver never receives the
interrupt, I added a small amount of debug to the Linux kernel source code
(in do_IRQ of irq.c) to see if the O/S is even seeing the interrupt. I
determined, without fail, that every time the kernel received our IRQ, it
issued an EOI and called our driver's interrupt service routine. I did this
by adding counters in the do_IRQ routine to count the # of my IRQs received,
# of times it was acknowledged and # of times a handler was called. If
everything has gone smoothly, all 3 of these should be the same - and they
are. Code snip:
asmlinkage unsigned int do_IRQ(struct pt_regs regs)
{
. . .
int irq_hit = 0; /* new, for debugging */
if (irq==kdbug_irq) /* kdbug_irq set by a user utility to turn this
debugging on */
{
irq_hit = 1;
atomic_inc(&kdbug_irq_cnt); /* new, for debugging */
}
#ifdef CONFIG_DEBUG_STACKOVERFLOW
. . .
spin_lock(&desc->lock);
desc->handler->ack(irq);
if (irq_hit) atomic_inc(&kdbug_irq_ack); /* new, for debugging */
/* REPLAY stuff . . . */
for (;;) {
spin_unlock(&desc->lock);
handle_IRQ_event(irq,®s, action);
spin_lock(&desc->lock);
if (irq_hit) /* new, for debugging */
atomic_inc(&kdbug_call_isr);
. . .
I then have a user application which periodically calls another driver that
calls another kernel debug routine which simply reports back the value of
these 3 counters and then re-inits them to zero. As I said, these counters
are always equal. Once the system "hangs" these counters are always zero,
so I concluded that Linux was never notified of the interrupt. I then added
code to peek at the 8259 registers to determine if it is receiving
notification of the pending interrupt. I discovered that the IMR register
did not have my IRQ, the keyboard, PS/2 mouse or ethernet interrupts masked
off. In addition, the IRR register shows that there is no interrupt pending
for our IRQ. The ISR register does not show evidence of our IRQ being
serviced either. However, the ISR shows that it is servicing the PS/2 mouse
interrupt. It appears to be "stuck" somewhere between the time of the
interrupt acknowledge cycle and it's driver issuing the EOI (since issuing
an EOI is what clears the bit in 8259's ISR register). Subsequent reads of
the 8259's ISR shows that the mouse is still being serviced. We also tried
a USB mouse to eliminate a mouse driver problem. We still experienced a
system "hang". It's almost as if something in the motherboard is is some
strange state because no other IRQs can get through. We have tried BIOS
updates and still have the problem. We have also used our software and
hardware successfully on other PC platforms using Linux (it was a Pentium
III system) and SUN platforms running Solaris. (Note that for this
debugging, we have configured the system so that our board uses IRQ 9
exclusively to eliminate any problems with interrupt sharing.) We also
tried a second system using an P4PE motherboard to insure that it wasn't a
single faulty motherboard. We received the same results.
While it appears to be hardware related, I wanted to see if anyone else has
experienced anything like this. My concern was that I might be experiencing
a race condition with the driver and board. I believe I have taken the
appropriate precautions that I have read about on this mailing list, but I
invite a second opinion (or third, forth, . . ) on the implementation.
Here's how it goes:
<code snip> from the driver's write routine:
. . .
/* The Reg element previously set using ioremap on the resource returned by
pci_resource_start */
writel(cpu_tole32(num_bytes), &pslState->Reg->DMA_Length);
wmb();
writel(cpu_to_le32(data), &pslState->Reg->DMA_Address);
/* start timer */
dmatimer.expires = jiffies + 0.5*HZ;
dmatimer.function = (void *) &dma_timeout;
dmatimer.data = (int) pslState;
add_timer(&dmatimer);
pslState->DMAIntr = INTR_PEND;
writel(BITS_TO_ENABLE_DMA, &pslState->Reg->ControlOrStatus);
wmb();
wait_event_interruptible(pslState->DMAIntrcvQ,
((pslState->DMAIntr==INTR_RECVD) || (pslState->DMAIntr==INTR_TO)));
del_timer(&dmatimer);
/* If our timer expired, then disable the DMA interrupt (since we never
entered our ISR to do this) */
if (pslState->DMAIntr == INTR_TO)
{
writel(BITS_TO_DISABLE_DMA, &pslState->Reg->ControlOrStatus);
wmb();
}
/* Error checking with boards's DMA reset and re-try logic goes here */
. . . < end code snip of write routine>
/* timeout routine called for the DMA timer */
void dma_timeout(STATE *pslState);
{
pslState->DMAIntr=INTR_TO;
wake_up_interruptible(&pslState->DMAIntrcvQ);
}
/* Our ISR code snip */
. . .
rval = (readl(&pslState(&pslState->Reg->ControlOrStatus) & DMA_INTERRUPT);
rmb();
if (rval)
{
writel(BITS_TO_DISABLE_DMA, &pslState->Reg->ControlOrStatus);
wmb();
pslState->DMAIntr=INTR_RECVD;
wake_up_interruptible(&pslState->DMAIntrcvQ);
}
I appreciate your feedback on whether you suspect any race conditions or why
the 8259 does not receive the interrupt. If you need any additional
information, please let me know! Thanks in advance for your help!
Kathy Frazier
Senior Software Engineer
Max Daetwyler Corporation-Dayton Division
2133 Lyons Road
Miamisburg, OH 45342
Tel #: 937.439-1582 ext 6158
Fax #: 937.439-1592
Email: [email protected]
http://www.daetwyler.com
Thanks for your reply, Andi.
>> We have a proprietary PCI board installed in a (UP) system with an ASUS
P4PE
>> motherboard (uses Intel 845PE chipset). This system is running Red Hat
9.0
>Have you checked the 845 errata sheets on the Intel website?
>Perhaps it is some known hardware bug.
>One thing you could try is to use Local APIC / IO APIC interrupt processing
>instead of 8259.
Our hardware engineer has combed the Intel and ASUS websites, but found
nothing. I'll give the APIC a try and see if I get different results and
let you know.
>>
>> /* start timer */
>> dmatimer.expires = jiffies + 0.5*HZ;
>That's a serious bug. You cannot use floating point in the kernel.
>It will corrupt the FP state of the user process.
HZ on the INTEL platform is 100, so this should simply add 50 to the current
value of jiffies. Besides, assigning the value to the unsigned int field
(expires) will truncate it to an integer anyway. Is there a more
appropriate way to handle a short timeout?
Thanks,
Kathy
On Tue, 15 Jul 2003, Kathy Frazier wrote:
> Thanks for your reply, Andi.
>
> >> We have a proprietary PCI board installed in a (UP) system with an ASUS
> P4PE
> >> motherboard (uses Intel 845PE chipset). This system is running Red Hat
> 9.0
>
> >Have you checked the 845 errata sheets on the Intel website?
> >Perhaps it is some known hardware bug.
>
> >One thing you could try is to use Local APIC / IO APIC interrupt processing
> >instead of 8259.
>
> Our hardware engineer has combed the Intel and ASUS websites, but found
> nothing. I'll give the APIC a try and see if I get different results and
> let you know.
>
> >>
> >> /* start timer */
> >> dmatimer.expires = jiffies + 0.5*HZ;
>
> >That's a serious bug. You cannot use floating point in the kernel.
> >It will corrupt the FP state of the user process.
>
> HZ on the INTEL platform is 100, so this should simply add 50 to the current
> value of jiffies. Besides, assigning the value to the unsigned int field
> (expires) will truncate it to an integer anyway. Is there a more
> appropriate way to handle a short timeout?
It will truncate it at runtime.
I suggest
dmatimer.expires = jiffies + HZ/2;
Also, you you need to use the time_before() macro, like
while ( time_before(jiffies, dmatimer.expires) )
do_something();
... so you don't have a timer-wrap problem during jiffie rollover if
you haven't already done so (I don't have your source here).
>
> Thanks,
> Kathy
Cheers,
Dick Johnson
Penguin : Linux version 2.4.20 on an i686 machine (797.90 BogoMips).
Why is the government concerned about the lunatic fringe? Think about it.
>There is a possibility that you are trying to use a shared interrupt,
>but the IRQ is set up for edge. You can only share level interrupts.
I checked it out. The Edge/Level Triggered Register (ELCR) of Intel's 82801
ICH4 PIC register shows that our IRQ is setup for edge.
>Also, in the keyboard ISR, there is code that will spin <forever>
>if the bit that was supposed to be true when the interrupt occurred,
>remains false. This means that, if the timing was screwed up on some
>feature-board bus interface state-machine, it could corrupt what
>other devices see on the bus. The result could be a "hang forever"
>condition.
Hmmm, this is good information. Exactly what bit are you referring too? Is
this in /drivers/char/keyboard.c? I took a quick look through, but didn't
see anything. With the symptoms I am experiencing, I thought it might be
due to a piece of software spinning in a loop after it issued a cli.
However, I should at least see any pending interrupts in the IRR of the
8259! The cli would just prevent the CPU from being notified of the pending
interrupt.
>The 8259 is such an old device that all its bugs are known and
>work-arounds have been in-place for ages. It is unlikely that your
>problem has anything to do with either in 8259 or existing kernel
>code. My bet's on with a level/edge problem or bus corruption due
>to bus-contention (timing issues).
I tend to agree with you - the 8259 has been around forever. The snooping
I've done with my debug indicates that that the IRQ never gets through from
the hardware. The CPU is still running software and apparantly other things
are getting scheduled. But no other IRQs are getting through! Something is
corrupted on the bus. This posting was a final sanity check to insure I
wasn't totally out in left field with respect to the driver software. I
know there are issues of race conditions when using the sleep_on/wake_up
mechanisms. However, from what I've read, wait_event_interruptible() method
will avoid this . . . Any thoughts on that?
Thanks for your input!
Kathy
Is Jeff Garzik's SCSI ATA driver going in 2.4.22? I've been using it
with great success with 2.4.21-ac4, but I haven't seen it in any
of the 2.4.22-pre kernels.
If it's not going in, is there an alternative for accessing serial
ATA devices in native/enhanced mode rather than legacy mode?
By the way, if you name your child libata, some say the name has
some nice qualities:
http://www.kabalarians.com/male/libata.htm
jeff
On Tue, Jul 15, 2003 at 08:58:16AM -0700, Jeff Mock wrote:
>
> Is Jeff Garzik's SCSI ATA driver going in 2.4.22? I've been using it
> with great success with 2.4.21-ac4, but I haven't seen it in any
> of the 2.4.22-pre kernels.
>
> If it's not going in, is there an alternative for accessing serial
> ATA devices in native/enhanced mode rather than legacy mode?
My preference would be to merge into 2.6.0-test first, then send it to
Marcelo after that. Otherwise, people upgrading 2.4->2.6 would get
shafted.
Besides that, I certainly would not object to it being in 2.4.22.
(but I'm biased :))
> By the way, if you name your child libata, some say the name has
> some nice qualities:
>
> http://www.kabalarians.com/male/libata.htm
That rules ;-)
Jeff
Kathy Frazier wrote:
> Thanks for your reply, Andi.
>
>
>>>We have a proprietary PCI board installed in a (UP) system with an ASUS
>
> P4PE
>
>>>motherboard (uses Intel 845PE chipset). This system is running Red Hat
>
> 9.0
>
>
>>Have you checked the 845 errata sheets on the Intel website?
>>Perhaps it is some known hardware bug.
>
>
>>One thing you could try is to use Local APIC / IO APIC interrupt processing
>>instead of 8259.
>
>
> Our hardware engineer has combed the Intel and ASUS websites, but found
> nothing. I'll give the APIC a try and see if I get different results and
> let you know.
>
>
>>>/* start timer */
>>>dmatimer.expires = jiffies + 0.5*HZ;
>
>
>>That's a serious bug. You cannot use floating point in the kernel.
>>It will corrupt the FP state of the user process.
>
>
> HZ on the INTEL platform is 100, so this should simply add 50 to the current
> value of jiffies. Besides, assigning the value to the unsigned int field
> (expires) will truncate it to an integer anyway.
Use HZ/2 instead. GCC doesn't optimize floating point constants to the
same degree it does integers, because it doesn't know what mode
(rounding, precision) the FPU is in.
--
Brian Gerst
Brian Gerst wrote:
> >>>dmatimer.expires = jiffies + 0.5*HZ;
> >>
> >>That's a serious bug. You cannot use floating point in the kernel.
> >>It will corrupt the FP state of the user process.
> >
> >HZ on the INTEL platform is 100, so this should simply add 50 to the
> >current
> >value of jiffies. Besides, assigning the value to the unsigned int field
> >(expires) will truncate it to an integer anyway.
>
> Use HZ/2 instead. GCC doesn't optimize floating point constants to the
> same degree it does integers, because it doesn't know what mode
> (rounding, precision) the FPU is in.
Optimising this to an integer add of 50 would be incorrect anyway.
Think about jiffies == 0xfffffffe.
The first statement is equivalent to:
dmatimer.expires = (unsigned long) ((double) + 50)
and would set dmatimer.expires to 0xffffffff.
Whereas the HZ/2 form is correct!
Have a nice day,
-- Jamie
On Tue, Jul 15, 2003 at 01:06:19PM -0400, Brian Gerst wrote:
> Use HZ/2 instead. GCC doesn't optimize floating point constants to the
> same degree it does integers, because it doesn't know what mode
> (rounding, precision) the FPU is in.
Isn't (HZ >> 1) better?
Regards,
Luciano Rocha
Luciano Miguel Ferreira Rocha wrote:
> On Tue, Jul 15, 2003 at 01:06:19PM -0400, Brian Gerst wrote:
>
>>Use HZ/2 instead. GCC doesn't optimize floating point constants to the
>>same degree it does integers, because it doesn't know what mode
>>(rounding, precision) the FPU is in.
>
>
> Isn't (HZ >> 1) better?
Same thing. GCC knows that division by a power of 2 is just a shift.
--
Brian Gerst
On Maw, 2003-07-15 at 19:24, Brian Gerst wrote:
> Luciano Miguel Ferreira Rocha wrote:
> > On Tue, Jul 15, 2003 at 01:06:19PM -0400, Brian Gerst wrote:
> >
> >>Use HZ/2 instead. GCC doesn't optimize floating point constants to the
> >>same degree it does integers, because it doesn't know what mode
> >>(rounding, precision) the FPU is in.
> >
> >
> > Isn't (HZ >> 1) better?
>
> Same thing. GCC knows that division by a power of 2 is just a shift.
Only for unsigned
Alan Cox wrote:
> > > Isn't (HZ >> 1) better?
> >
> > Same thing. GCC knows that division by a power of 2 is just a shift.
>
> Only for unsigned
Or when HZ is constant :)
-- J?amie
Hi Alan.
>>>> Use HZ/2 instead. GCC doesn't optimize floating point constants
>>>> to the same degree it does integers, because it doesn't know what
>>>> mode (rounding, precision) the FPU is in.
>>> Isn't (HZ >> 1) better?
>> Same thing. GCC knows that division by a power of 2 is just a shift.
> Only for unsigned
When I did the "Assembler Language Maths Logic" module for my degree,
we learned that processors used SRL (Shift Right Logical) to divide
unsigned numbers by powers of 2, and SRA (Shift Right Arithmetic) to
divide signed numbers by powers of 2. Can't GCC handle that?
Best wishes from Riley.
---
* Nothing as pretty as a smile, nothing as ugly as a frown.
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.502 / Virus Database: 300 - Release Date: 18-Jul-2003
On Sun, 2003-07-20 at 14:04, Riley Williams wrote:
> When I did the "Assembler Language Maths Logic" module for my degree,
> we learned that processors used SRL (Shift Right Logical) to divide
> unsigned numbers by powers of 2, and SRA (Shift Right Arithmetic) to
> divide signed numbers by powers of 2. Can't GCC handle that?
Sure it can, but >>1 isn't the same as /2 for signed numbers.
(Hint: -1 / 2 != -1)
J