We have a proprietary application that is an emulation of an old real-time os
(MPX-32). We have a couple of pieces of hardware (pci cards) with soon to be GPL
drivers that go along with the emulation package and the hardware. We are using
a 2.4.18 kernel with the O(1) scheduler and a couple of other patches. One of
these pieces of hardware is a pci card with 6 each 1us resolution timers and 8
external interrupts (4 in and 4 out). We have fortran a program on the old
real-time os that tells us about the interrupt latency and determinism (jitter).
When we run this program in the emulation on an intel box all is well. When we
run this in the emulation on an AMD MP 1900+ box the determinism (jitter) is
very bad. Sometimes as much as 500us. On the Dual Intel 2.2 p4 zeon the
determinism (jitter) is under 50us. All the other benchmarks we run under the
emulation tell us that the AMD box is the faster box. It's also cheaper. So I
guess my question is, are there any known problem with AMD's and interrupt
latency jitter. I might also add that the only way we get satisfactory numbers,
even on the P4 zeons, was with the affinity calls in the O(1) scheduler along
with the affinitization of the pci cards' irq. We basically create a pset for
our app using these affinity calls then do the same with the irqs down in the
driver for this pci card. So our emulation is run in one processor and all else
(linux) in the other. We do the same for all the irqs. Why does this only give
good results on intel and not amd??? We have tested many intel MB/P4 combos but
only one AMD, a Tyan s2466n.
Sorry if I'm wasting your time
Thanks and regards
Mark
On Fri, 25 Oct 2002, Mark Hounschell wrote:
> We have a proprietary application that is an emulation of an old real-time os
> (MPX-32). We have a couple of pieces of hardware (pci cards) with soon to be GPL
> drivers that go along with the emulation package and the hardware. We are using
> a 2.4.18 kernel with the O(1) scheduler and a couple of other patches. One of
[SNIPPED....]
Please use the [Enter] key. Lines should have "\n" and not auto-wrap
in your mailer.
If you are measuring the interrupt latency jitter, you
must disconnect your Ethernet wire if you have a Bus Mastering
(read PCI) Ethernet board. You also have to make sure that
no other Bus Masters are able to run during the measurements.
This is because the Bus Masters will keep the CPU(s) off the bus
for variable lengths of time as they transfer variable lengths
of data. This gives you the jitter.
Even though you may not have any connections to your machine,
M$ on LANs generate much broadcast traffic that your network
software has to read, then drop on the floor.
What this means, frankly, is that if interrupt latency is
in your specification, you can't use any Bus Mastering devices.
It's just that simple.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Bush : The Fourth Reich of America
On Fri, 2002-10-25 at 20:40, Mark Hounschell wrote:
> When we run this program in the emulation on an intel box all is well. When we
> run this in the emulation on an AMD MP 1900+ box the determinism (jitter) is
> very bad. Sometimes as much as 500us. On the Dual Intel 2.2 p4 zeon the
> determinism (jitter) is under 50us. All the other benchmarks we run under the
> emulation tell us that the AMD box is the faster box. It's also cheaper. So I
> guess my question is, are there any known problem with AMD's and interrupt
> latency jitter. I might also add that the only way we get satisfactory numbers,
A lot of the IRQ delivery depends on the APIC. SO for example you'd
probably see horrible numbers on a PIII, but very good on PIV. It also
depends on chipset vagueries. You might also need to check that your PCI
behaviour has no posting errors since the AMD certainly seems to do a
lot more aggressive PCI posting.
Does "noapic" change the jitter ?
"Richard B. Johnson" wrote:
>
> [SNIPPED....]
> Please use the [Enter] key. Lines should have "\n" and not auto-wrap
> in your mailer.
Sorry...
>
> If you are measuring the interrupt latency jitter, you
> must disconnect your Ethernet wire if you have a Bus Mastering
> (read PCI) Ethernet board. You also have to make sure that
> no other Bus Masters are able to run during the measurements.
>
> This is because the Bus Masters will keep the CPU(s) off the bus
> for variable lengths of time as they transfer variable lengths
> of data. This gives you the jitter.
>
> Even though you may not have any connections to your machine,
> M$ on LANs generate much broadcast traffic that your network
> software has to read, then drop on the floor.
>
> What this means, frankly, is that if interrupt latency is
> in your specification, you can't use any Bus Mastering devices.
> It's just that simple.
>
That makes sense. But, both these Intel and AMD boxes have pretty much the same
config as far as pci cards and pci busses. They both have 1 or 2 66mhz and a 33
mhz bus.
The Intel box used right now is a Super-micro p4dc6+ and only has our 2 33mhz
cards in it. It has on board UW-scsi-2 controller using a 66MHz bus where as the
AMD has no controllers on the 66mhz bus. It is using the onboard IDE controller.
The intel has built in network card that IS active when running the emulation
and the AMD has a 3c905 card that is also active. Other than that they are the
same. I thought all recent pci cards were bus mastering capable these days??
Regards
Mark
Alan Cox wrote:
>
> On Fri, 2002-10-25 at 20:40, Mark Hounschell wrote:
> > When we run this program in the emulation on an intel box all is well. When we
> > run this in the emulation on an AMD MP 1900+ box the determinism (jitter) is
> > very bad. Sometimes as much as 500us. On the Dual Intel 2.2 p4 zeon the
> > determinism (jitter) is under 50us. All the other benchmarks we run under the
> > emulation tell us that the AMD box is the faster box. It's also cheaper. So I
> > guess my question is, are there any known problem with AMD's and interrupt
> > latency jitter. I might also add that the only way we get satisfactory numbers,
>
> A lot of the IRQ delivery depends on the APIC. SO for example you'd
> probably see horrible numbers on a PIII, but very good on PIV. It also
> depends on chipset vagueries. You might also need to check that your PCI
> behaviour has no posting errors since the AMD certainly seems to do a
> lot more aggressive PCI posting.
>
> Does "noapic" change the jitter ?
Actually if I use noapci then the irq affinity stuff does not work, which in
itself causes
the jitter to be unexceptable. On the AMD, the process and irq affinity appears
to be working
correctly, according to /proc/interrupts, but maybe it's really not??? Thats
what it acts
like. It acts just like if I hadn't set the affinity for the emulation and irq
on the P4,
just not as bad.
Regards
Mark
On Fri, 2002-10-25 at 21:45, Mark Hounschell wrote:
> That makes sense. But, both these Intel and AMD boxes have pretty much the same
> config as far as pci cards and pci busses. They both have 1 or 2 66mhz and a 33
> mhz bus.
> The Intel box used right now is a Super-micro p4dc6+ and only has our 2 33mhz
> cards in it. It has on board UW-scsi-2 controller using a 66MHz bus where as the
> AMD has no controllers on the 66mhz bus. It is using the onboard IDE controller.
> The intel has built in network card that IS active when running the emulation
> and the AMD has a 3c905 card that is also active. Other than that they are the
> same. I thought all recent pci cards were bus mastering capable these days??
Pretty much. IDE and graphics cards tend to be badly behaved at times
but the other stuff ought to behave better. The AMD does have some
tunables about fairness/performance etc in the chipset. I wonder which
way the vendor sets them