Hi,
some time ago (2.4.2 time), I created a patch that allows using a
multiprocessor system with different speed ix86 CPUs and Linux.
The patch does the following:
* Make sure we got the flags right (in case they are different) before we
enable XMM/FXSR/.... Right means common subset of supported features.
* Make cpu_khz a per CPU field and use it in the fast_get_timeofday(). The
offset from last timer tick also must be per CPU.
The patch works fine, but it has a limited scope: It does not try to teach
the scheduler about the fact that the CPUs are different. So a process that
runs on the slower CPU does not get any bonus. It's just unlucky ...
(Some dynamic prio weighting with the cpu_khz should not be too hard in the
goodness calc, but I wanted to keep things simple.)
I attach the patch against 2.4.16.
Chandra, this is what you've been looking for, right?
Enjoy!
--
Kurt Garloff <[email protected]> Eindhoven, NL
GPG key: See mail header, key servers Linux kernel development
SuSE Linux AG, Nuernberg, DE SCSI, Security
On Mon, Mar 11, 2002 at 04:34:21AM +0100, Kurt Garloff wrote:
> @@ -53,6 +54,10 @@
> unsigned long *pmd_quick;
> unsigned long *pte_quick;
> unsigned long pgtable_cache_sz;
> + unsigned long cpu_khz;
> + unsigned long fast_gettimeoffset_quotient;
> + unsigned long last_tsc_low;
> + struct timeval xtime;
> } __attribute__((__aligned__(SMP_CACHE_BYTES)));
>
> #define X86_VENDOR_INTEL 0
the only problem is if you happen to get the timer irq always in the
same cpu for a few seconds, then the last_tsc_low will wrap around and
gettimeofday will be wrong. And even if you snapshot the full 64bit of the
tsc you'll run into some trouble if the timer irq will be delivered only
to the same cpu for a long time (for example if you use irq bindings).
you'd lose precision and you'll run into the measuration errors of
fast_gettimeoffset_quotient. The right support for asynchronous TSC
handling is a bit more complicated unfortunately.
Andrea
Hi Andrea,
On Mon, Mar 11, 2002 at 05:29:54AM +0100, Andrea Arcangeli wrote:
> the only problem is if you happen to get the timer irq always in the
> same cpu for a few seconds, then the last_tsc_low will wrap around and
> gettimeofday will be wrong. And even if you snapshot the full 64bit of the
> tsc you'll run into some trouble if the timer irq will be delivered only
> to the same cpu for a long time (for example if you use irq bindings).
> you'd lose precision and you'll run into the measuration errors of
> fast_gettimeoffset_quotient. The right support for asynchronous TSC
> handling is a bit more complicated unfortunately.
If your APIC works, your CPUs should get the timer IRQs in alternating order.
At least that is what seems to happen on the SMP box where I created and
tested this patch. It works perfectly there.
In the more general case, you may need to do more, right.
You could make the xtime global again and send IPIs for every timer
interrupt to make all CPUs update their TSC offset.
Storing the full 64bit TSC may be a good option though.
So, if people wanted to have this included I'd rather document the
limitations (don't bind the timer IRQ to one CPU) than introducing
complexity which may hurt the normal SMP user.
Regards,
--
Kurt Garloff <[email protected]> Eindhoven, NL
GPG key: See mail header, key servers Linux kernel development
SuSE Linux AG, Nuernberg, DE SCSI, Security
On Mon, Mar 11, 2002 at 12:25:49PM +0100, Kurt Garloff wrote:
> Hi Andrea,
>
> On Mon, Mar 11, 2002 at 05:29:54AM +0100, Andrea Arcangeli wrote:
> > the only problem is if you happen to get the timer irq always in the
> > same cpu for a few seconds, then the last_tsc_low will wrap around and
> > gettimeofday will be wrong. And even if you snapshot the full 64bit of the
> > tsc you'll run into some trouble if the timer irq will be delivered only
> > to the same cpu for a long time (for example if you use irq bindings).
> > you'd lose precision and you'll run into the measuration errors of
> > fast_gettimeoffset_quotient. The right support for asynchronous TSC
> > handling is a bit more complicated unfortunately.
>
> If your APIC works, your CPUs should get the timer IRQs in alternating order.
Maybe I remeber wrong, but AFIK the io-apic isn't required to scale the
irq load in alternating order, it is perfectly allowed to deliver the
irq always to the same cpu for several seconds. I know the probability
for that to happen is low but it can happen.
Andrea
Andrea Arcangeli <[email protected]> writes:
> On Mon, Mar 11, 2002 at 12:25:49PM +0100, Kurt Garloff wrote:
> > Hi Andrea,
> >
> > On Mon, Mar 11, 2002 at 05:29:54AM +0100, Andrea Arcangeli wrote:
> > > the only problem is if you happen to get the timer irq always in the
> > > same cpu for a few seconds, then the last_tsc_low will wrap around and
> > > gettimeofday will be wrong. And even if you snapshot the full 64bit of the
> > > tsc you'll run into some trouble if the timer irq will be delivered only
> > > to the same cpu for a long time (for example if you use irq bindings).
> > > you'd lose precision and you'll run into the measuration errors of
> > > fast_gettimeoffset_quotient. The right support for asynchronous TSC
> > > handling is a bit more complicated unfortunately.
> >
> > If your APIC works, your CPUs should get the timer IRQs in alternating order.
>
> Maybe I remeber wrong, but AFIK the io-apic isn't required to scale the
> irq load in alternating order, it is perfectly allowed to deliver the
> irq always to the same cpu for several seconds. I know the probability
> for that to happen is low but it can happen.
Actually I know of at least one dual P4 Xeon board where I haven't seen anything
except IPI go to the second cpu.
Eric
On Mon, Mar 11, 2002 at 04:34:21AM +0100, Kurt Garloff wrote:
> Hi,
>
> some time ago (2.4.2 time), I created a patch that allows using a
> multiprocessor system with different speed ix86 CPUs and Linux.
>
> The patch does the following:
> * Make sure we got the flags right (in case they are different) before we
> enable XMM/FXSR/.... Right means common subset of supported features.
Running quasi symetric system (dual P-II 300 MHz, but different cores, one
is Klamath, other is Deschutes) the fix for the flags is very usefull and
I'd like to see it integrated in the stock kernels. Perhaps the flags and
the (more controversial) speed patches can be split?
Regards,
Frank.
--
+---- --- -- - - - -
| Frank van de Pol -o) A-L-S-A
| [email protected] /\\ Sounds good!
| http://www.alsa-project.org _\_v
| Linux - Why use Windows if we have doors available?
> > irq always to the same cpu for several seconds. I know the probability
> > for that to happen is low but it can happen.
>
> Actually I know of at least one dual P4 Xeon board where I haven't seen anything
> except IPI go to the second cpu.
Expect that to occur. The random distribution stuff doesn't seem to be a
feature of all pentium IV systems. Ie this bug does want fixing
>> Actually I know of at least one dual P4 Xeon board where I
>> haven't seen anything except IPI go to the second cpu.
>
> Expect that to occur. The random distribution stuff doesn't seem to be a
> feature of all pentium IV systems. Ie this bug does want fixing
Dave Olien published a patch which will make this issue much
better for P4 - setting the TPR so we route interrupts according
to how important the processors current work is. Should alleviate
the "dump everything on one CPU" P4ness.
Would still be nice to fix this though.
M.