It's been known for quite some time that the TSCs are not synced between
cores on Athlon X2 machines and this screws up the kernel's timekeeping,
as it still uses the TSC as the default time source on these machines.
This problem still seems to be present in the latest kernels. What is
the plan to fix it? Is the fix simply to make the kernel use the ACPI
PM timer by default on Athlon X2?
Lee
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of Lee Revell
> Sent: Thursday, January 12, 2006 4:18 PM
> To: linux-kernel
> Subject: Dual core Athlons and unsynced TSCs
>
> It's been known for quite some time that the TSCs are not
> synced between cores on Athlon X2 machines and this screws up
> the kernel's timekeeping, as it still uses the TSC as the
> default time source on these machines.
>
> This problem still seems to be present in the latest kernels.
> What is the plan to fix it? Is the fix simply to make the
> kernel use the ACPI PM timer by default on Athlon X2?
Do we know if this also affects dual-core opterons?
The symptoms are that the clocks run at 2x the speed, correct?
Roger
On Fri, 2006-01-13 at 09:10 -0600, Roger Heflin wrote:
>
> > -----Original Message-----
> > From: [email protected]
> > [mailto:[email protected]] On Behalf Of Lee Revell
> > Sent: Thursday, January 12, 2006 4:18 PM
> > To: linux-kernel
> > Subject: Dual core Athlons and unsynced TSCs
> >
> > It's been known for quite some time that the TSCs are not
> > synced between cores on Athlon X2 machines and this screws up
> > the kernel's timekeeping, as it still uses the TSC as the
> > default time source on these machines.
> >
> > This problem still seems to be present in the latest kernels.
> > What is the plan to fix it? Is the fix simply to make the
> > kernel use the ACPI PM timer by default on Athlon X2?
>
>
> Do we know if this also affects dual-core opterons?
>
> The symptoms are that the clocks run at 2x the speed, correct?
No, worse. The monotonic clock can go backwards. The tscs of the CPUs
are not in sync when one slows down due to idle. So if you read from
two different CPUs, you may get the second read have an earlier time
than the first. Breaks the rule of what a monotonic clock is.
-- Steve
On Fri, 2006-01-13 at 11:04 -0500, Steven Rostedt wrote:
> On Fri, 2006-01-13 at 09:10 -0600, Roger Heflin wrote:
> >
> > > -----Original Message-----
> > > From: [email protected]
> > > [mailto:[email protected]] On Behalf Of Lee Revell
> > > Sent: Thursday, January 12, 2006 4:18 PM
> > > To: linux-kernel
> > > Subject: Dual core Athlons and unsynced TSCs
> > >
> > > It's been known for quite some time that the TSCs are not
> > > synced between cores on Athlon X2 machines and this screws up
> > > the kernel's timekeeping, as it still uses the TSC as the
> > > default time source on these machines.
> > >
> > > This problem still seems to be present in the latest kernels.
> > > What is the plan to fix it? Is the fix simply to make the
> > > kernel use the ACPI PM timer by default on Athlon X2?
> >
> >
> > Do we know if this also affects dual-core opterons?
> >
> > The symptoms are that the clocks run at 2x the speed, correct?
>
> No, worse. The monotonic clock can go backwards. The tscs of the CPUs
> are not in sync when one slows down due to idle. So if you read from
> two different CPUs, you may get the second read have an earlier time
> than the first. Breaks the rule of what a monotonic clock is.
>
Steve,
I don't have hardware to test this, can you confirm that the only
workaround needed is to boot with "clock=pmtmr"?
Lee
On Fri, 13 Jan 2006, Lee Revell wrote:
> I don't have hardware to test this, can you confirm that the only
> workaround needed is to boot with "clock=pmtmr"?
For which kernel?
-- Steve
On Fri, 2006-01-13 at 12:52 -0500, Steven Rostedt wrote:
> On Fri, 13 Jan 2006, Lee Revell wrote:
>
> > I don't have hardware to test this, can you confirm that the only
> > workaround needed is to boot with "clock=pmtmr"?
>
> For which kernel?
I believe the problem exists on all kernels, it should not matter.
Lee
On Thu, Jan 12, 2006 at 05:17:39PM -0500, Lee Revell wrote:
> It's been known for quite some time that the TSCs are not synced between
> cores on Athlon X2 machines and this screws up the kernel's timekeeping,
> as it still uses the TSC as the default time source on these machines.
>
> This problem still seems to be present in the latest kernels. What is
> the plan to fix it? Is the fix simply to make the kernel use the ACPI
> PM timer by default on Athlon X2?
If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET
timer instead. It doesn't solve the problem for apps or other kernel bits
that use rdtsc directly, but there are other fixes for that floating
around which will hopefully get consideration when they're mature.
On Fri, 2006-01-13 at 10:06 -0800, [email protected] wrote:
> On Thu, Jan 12, 2006 at 05:17:39PM -0500, Lee Revell wrote:
> > It's been known for quite some time that the TSCs are not synced between
> > cores on Athlon X2 machines and this screws up the kernel's timekeeping,
> > as it still uses the TSC as the default time source on these machines.
> >
> > This problem still seems to be present in the latest kernels. What is
> > the plan to fix it? Is the fix simply to make the kernel use the ACPI
> > PM timer by default on Athlon X2?
>
> If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET
> timer instead. It doesn't solve the problem for apps or other kernel bits
> that use rdtsc directly, but there are other fixes for that floating
> around which will hopefully get consideration when they're mature.
>
The apps can be converted to use clock_gettime(), but that does not help
if the kernel can't keep reasonable time on these machines.
But if we just use "clock=pmtmr" by default on these machines the TSC is
not a problem right?
Lee
On Fri, Jan 13, 2006 at 12:58:36PM -0500, Lee Revell wrote:
> > If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET
> > timer instead. It doesn't solve the problem for apps or other kernel bits
> > that use rdtsc directly, but there are other fixes for that floating
> > around which will hopefully get consideration when they're mature.
>
> The apps can be converted to use clock_gettime(), but that does not help
> if the kernel can't keep reasonable time on these machines.
Some apps/users need higher resolution and lower overhead that only rdtsc
can offer currently.
> But if we just use "clock=pmtmr" by default on these machines the TSC is
> not a problem right?
I never tried it with pmtimer, we had HPET available. Empirically TSC did
not work (and we had a simple test case to show how bad it could get).
HPET made that go away for users of gettimeofday().
We're exploring rdtsc-compatible solutions.
On Fri, 2006-01-13 at 10:16 -0800, [email protected] wrote:
> On Fri, Jan 13, 2006 at 12:58:36PM -0500, Lee Revell wrote:
> > > If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET
> > > timer instead. It doesn't solve the problem for apps or other kernel bits
> > > that use rdtsc directly, but there are other fixes for that floating
> > > around which will hopefully get consideration when they're mature.
> >
> > The apps can be converted to use clock_gettime(), but that does not help
> > if the kernel can't keep reasonable time on these machines.
>
> Some apps/users need higher resolution and lower overhead that only rdtsc
> can offer currently.
>
But obviously if the TSC gives wildly inaccurate results, it cannot be
used no matter how low the overhead.
> > But if we just use "clock=pmtmr" by default on these machines the TSC is
> > not a problem right?
>
> I never tried it with pmtimer, we had HPET available. Empirically TSC did
> not work (and we had a simple test case to show how bad it could get).
> HPET made that go away for users of gettimeofday().
>
> We're exploring rdtsc-compatible solutions.
>
Since timekeeping on these machines has always been completely broken,
shouldn't the default time source have been changed to the PM timer or
HPET as soon as the problem was known?
Lee
On Fri, 13 Jan 2006, Lee Revell wrote:
> On Fri, 2006-01-13 at 12:52 -0500, Steven Rostedt wrote:
> > On Fri, 13 Jan 2006, Lee Revell wrote:
> >
> > > I don't have hardware to test this, can you confirm that the only
> > > workaround needed is to boot with "clock=pmtmr"?
> >
> > For which kernel?
>
> I believe the problem exists on all kernels, it should not matter.
Well, I just booted up (and am currently running) 2.6.15-rt4-sr2 and I
used "clock=pmtmr". I haven't got any
read_tsc: ACK! TSC went backward! Unsynced TSCs?
If that's what you want to know?
But the latency traces are still screwed up.
-- Steve
On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote:
> > Some apps/users need higher resolution and lower overhead that only rdtsc
> > can offer currently.
>
> But obviously if the TSC gives wildly inaccurate results, it cannot be
> used no matter how low the overhead.
unless we can re-sync the TSCs often enough that apps don't notice.
> > I never tried it with pmtimer, we had HPET available. Empirically TSC did
> > not work (and we had a simple test case to show how bad it could get).
> > HPET made that go away for users of gettimeofday().
> >
> > We're exploring rdtsc-compatible solutions.
> >
>
> Since timekeeping on these machines has always been completely broken,
> shouldn't the default time source have been changed to the PM timer or
> HPET as soon as the problem was known?
If you have HPET, the kernel will prefer that. I don't know enough about
the PMtimer overhead or resolution to say for sure.
On Fri, 2006-01-13 at 10:55 -0800, [email protected] wrote:
> On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote:
> > > Some apps/users need higher resolution and lower overhead that only rdtsc
> > > can offer currently.
> >
> > But obviously if the TSC gives wildly inaccurate results, it cannot be
> > used no matter how low the overhead.
>
> unless we can re-sync the TSCs often enough that apps don't notice.
>
You'd have to quantify that somehow, in terms of the max drift rate
(ppm), and the max resolution available (< tsc frequency).
Either that, or track an offset, and use one TSC as truth, and update
the correction factor for the other TSCs as often as needed, maybe?
This is kind of analogous to the "drift" NTP calculates against a
free-running oscillator.
So you'd be pushing that functionality deeper into the OS-core.
Dave Mills had that "hardpps" stuff in there for a while, it might be a
starting point.
Just some thoughts for now...
Sven
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
***********************************
Sven-Thorsten Dietrich
Real-Time Software Architect
MontaVista Software, Inc.
1237 East Arques Ave.
Sunnyvale, CA 94085
Phone: 408.992.4515
Fax: 408.328.9204
http://www.mvista.com
Platform To Innovate
***********************************
On Fri, 2006-01-13 at 13:41 -0500, Steven Rostedt wrote:
> On Fri, 13 Jan 2006, Lee Revell wrote:
>
> > On Fri, 2006-01-13 at 12:52 -0500, Steven Rostedt wrote:
> > > On Fri, 13 Jan 2006, Lee Revell wrote:
> > >
> > > > I don't have hardware to test this, can you confirm that the only
> > > > workaround needed is to boot with "clock=pmtmr"?
> > >
> > > For which kernel?
> >
> > I believe the problem exists on all kernels, it should not matter.
>
>
> Well, I just booted up (and am currently running) 2.6.15-rt4-sr2 and I
> used "clock=pmtmr". I haven't got any
>
> read_tsc: ACK! TSC went backward! Unsynced TSCs?
>
>
> If that's what you want to know?
I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually
guaranteed to be monotonic on these machines AKA do we break POSIX
compliance or not.
Lee
On Fri, 2006-01-13 at 10:56 -0800, Sven-Thorsten Dietrich wrote:
> On Fri, 2006-01-13 at 10:55 -0800, [email protected] wrote:
> > On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote:
> > > > Some apps/users need higher resolution and lower overhead that only rdtsc
> > > > can offer currently.
> > >
> > > But obviously if the TSC gives wildly inaccurate results, it cannot be
> > > used no matter how low the overhead.
> >
> > unless we can re-sync the TSCs often enough that apps don't notice.
> >
>
> You'd have to quantify that somehow, in terms of the max drift rate
> (ppm), and the max resolution available (< tsc frequency).
>
> Either that, or track an offset, and use one TSC as truth, and update
> the correction factor for the other TSCs as often as needed, maybe?
>
> This is kind of analogous to the "drift" NTP calculates against a
> free-running oscillator.
>
> So you'd be pushing that functionality deeper into the OS-core.
>
> Dave Mills had that "hardpps" stuff in there for a while, it might be a
> starting point.
>
> Just some thoughts for now...
>
It kind of makes you wonder what in the heck AMD were thinking, whether
they realized that this design decision would cause so many problems at
the OS level (it's broken at least Linux and Solaris). Maybe Windows
keeps time in a way that was unaffected by this?
Lee
On Fri, 2006-01-13 at 14:00 -0500, Lee Revell wrote:
> On Fri, 2006-01-13 at 10:56 -0800, Sven-Thorsten Dietrich wrote:
> > On Fri, 2006-01-13 at 10:55 -0800, [email protected] wrote:
> > > On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote:
> > > > > Some apps/users need higher resolution and lower overhead that only rdtsc
> > > > > can offer currently.
> > > >
> > > > But obviously if the TSC gives wildly inaccurate results, it cannot be
> > > > used no matter how low the overhead.
> > >
> > > unless we can re-sync the TSCs often enough that apps don't notice.
> > >
> >
> > You'd have to quantify that somehow, in terms of the max drift rate
> > (ppm), and the max resolution available (< tsc frequency).
> >
> > Either that, or track an offset, and use one TSC as truth, and update
> > the correction factor for the other TSCs as often as needed, maybe?
> >
> > This is kind of analogous to the "drift" NTP calculates against a
> > free-running oscillator.
> >
> > So you'd be pushing that functionality deeper into the OS-core.
> >
> > Dave Mills had that "hardpps" stuff in there for a while, it might be a
> > starting point.
> >
> > Just some thoughts for now...
> >
>
> It kind of makes you wonder what in the heck AMD were thinking, whether
> they realized that this design decision would cause so many problems at
> the OS level (it's broken at least Linux and Solaris). Maybe Windows
> keeps time in a way that was unaffected by this?
>
Last time I looked at Windows, we had to create an RT task to
interpolate between jiffies, b/c Windows was only keeping time at the
10 / 16 (smp) ms resolution. (to get NTP to run)
That was 6 years ago.
I haven't looked back since, and may I turn to salt or stone if I ever
do.
> Lee
>
--
***********************************
Sven-Thorsten Dietrich
Real-Time Software Architect
MontaVista Software, Inc.
1237 East Arques Ave.
Sunnyvale, CA 94085
Phone: 408.992.4515
Fax: 408.328.9204
http://www.mvista.com
Platform To Innovate
***********************************
Le vendredi 13 janvier 2006 ? 10:56 -0800, Sven-Thorsten Dietrich a
?crit :
> On Fri, 2006-01-13 at 10:55 -0800, [email protected] wrote:
> > On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote:
> > > > Some apps/users need higher resolution and lower overhead that only rdtsc
> > > > can offer currently.
> > >
> > > But obviously if the TSC gives wildly inaccurate results, it cannot be
> > > used no matter how low the overhead.
> >
> > unless we can re-sync the TSCs often enough that apps don't notice.
> >
>
> You'd have to quantify that somehow, in terms of the max drift rate
> (ppm), and the max resolution available (< tsc frequency).
>
> Either that, or track an offset, and use one TSC as truth, and update
> the correction factor for the other TSCs as often as needed, maybe?
As often as needed being each time a thread changes CPU ?
On Fri, Jan 13, 2006 at 10:56:13AM -0800, Sven-Thorsten Dietrich wrote:
> On Fri, 2006-01-13 at 10:55 -0800, [email protected] wrote:
> > unless we can re-sync the TSCs often enough that apps don't notice.
>
> You'd have to quantify that somehow, in terms of the max drift rate
> (ppm), and the max resolution available (< tsc frequency).
>
> Either that, or track an offset, and use one TSC as truth, and update
> the correction factor for the other TSCs as often as needed, maybe?
>
> This is kind of analogous to the "drift" NTP calculates against a
> free-running oscillator.
>
> So you'd be pushing that functionality deeper into the OS-core.
>
> Dave Mills had that "hardpps" stuff in there for a while, it might be a
> starting point.
>
> Just some thoughts for now...
There's some chatter here about a solution involving a lazy sync of TSCs
to the HPET (or other) whenever an app calls rdtsc after a potentially
unsyncing event.
For example, 'hlt' will initiate C1 which may cause clock ramping (and TSC
skew). We can trap rdtsc after a hlt and re-sync the TSCs to some truly
monotonic source, like HPET.
I don't have all the details, some problems remain, and the work is not
quite done yet, but it looks promising.
Even if we eventually get synced TSCs, it's too little too late.
basically, anything in-kernel that uses rdtsc is bound to break, and any
app that uses rdtsc had better be using CPU affinity.
Tim
On Fri, 2006-01-13 at 13:55 -0500, Lee Revell wrote:
> >
> > If that's what you want to know?
>
> I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually
> guaranteed to be monotonic on these machines AKA do we break POSIX
> compliance or not.
Nope it doesn't work :-(
I ran this test:
http://www.kihontech.com/tests/rt/monotonic.c
And got this:
$ ./monotonic
main program pid=8477
hello from thread 0!
hello from thread 1!
hello from thread 2!
hello from thread 3!
hello from thread 4!
Failed! prev: 6279.925610302 current: 6279.874615349
$ cat /proc/cmdline
root=/dev/md0 ro console=ttyS0,115200 console=tty0 nmi_watchdog=2 lapic earlyprintk=ttyS0,115200 clock=pmtmr
$ uname -r
2.6.15-rt4-sr2
(I also added Ingo, Thomas, and John in the CC)
I'll reboot to vanilla 2.6.15 and see if that is broken too (just to
make sure)
-- Steve
On Fri, 2006-01-13 at 15:09 -0500, Steven Rostedt wrote:
> Nope it doesn't work :-(
>
> I ran this test:
> http://www.kihontech.com/tests/rt/monotonic.c
>
[...]
>
> I'll reboot to vanilla 2.6.15 and see if that is broken too (just to
> make sure)
>
Failed on 2.6.15 too:
$ ./monotonic
main program pid=6724
hello from thread 0!
hello from thread 4!
Failed! prev: 316.952599248 current: 316.951899248
hello from thread 3!
hello from thread 2!
hello from thread 1!
$ uname -r
2.6.15
$ cat /proc/cmdline
root=/dev/md0 ro console=tty0 nmi_watchdog=2 lapic clock=pmtmr
:-(
-- Steve
On Fri, 2006-01-13 at 15:26 -0500, Steven Rostedt wrote:
> On Fri, 2006-01-13 at 15:09 -0500, Steven Rostedt wrote:
>
> > Nope it doesn't work :-(
> >
> > I ran this test:
> > http://www.kihontech.com/tests/rt/monotonic.c
> >
> [...]
> >
> > I'll reboot to vanilla 2.6.15 and see if that is broken too (just to
> > make sure)
> >
>
> Failed on 2.6.15 too:
That's what I suspected based on reports from several JACK users (JACK
used to use rdtsc directly, it now uses clock_gettime).
Hmm, it looks like the kernel uses the TSC left and right, even if
clock=pmtmr is used. There's even an rdtscll in pmtimer_mark_offset().
Wow, what a mess. Thanks, AMD...
Lee
On Fri, 2006-01-13 at 15:26 -0500, Steven Rostedt wrote:
> On Fri, 2006-01-13 at 15:09 -0500, Steven Rostedt wrote:
>
> > Nope it doesn't work :-(
> >
> > I ran this test:
> > http://www.kihontech.com/tests/rt/monotonic.c
> >
> [...]
> >
> > I'll reboot to vanilla 2.6.15 and see if that is broken too (just to
> > make sure)
> >
>
> Failed on 2.6.15 too:
>
Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
unconditionally.
Lee
On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote:
> Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
> unconditionally.
Can you try
http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch
please ?
tglx
On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote:
> On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote:
>
> > Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
> > unconditionally.
>
> Can you try
>
> http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch
>
> please ?
I can try it (well I can pass it on to someone who has the hardware),
but that's a huge patch, not likely to be mergeable in the 2.6.16
timeframe. Surely there has to be a way this can be fixed for 2.6.16?
Lee
On Fri, 2006-01-13 at 15:48 -0500, Lee Revell wrote:
> On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote:
> > On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote:
> >
> > > Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
> > > unconditionally.
> >
> > Can you try
> >
> > http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch
> >
> > please ?
>
> I can try it (well I can pass it on to someone who has the hardware),
> but that's a huge patch, not likely to be mergeable in the 2.6.16
> timeframe. Surely there has to be a way this can be fixed for 2.6.16?
Parts of the patch are in mainline (-gitX), but those alone will not fix
the problem AFAICS
tglx
On Fri, 13 Jan 2006, Lee Revell wrote:
> On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote:
> > On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote:
> >
> > > Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
> > > unconditionally.
> >
> > Can you try
> >
> > http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch
> >
> > please ?
>
> I can try it (well I can pass it on to someone who has the hardware),
> but that's a huge patch, not likely to be mergeable in the 2.6.16
> timeframe. Surely there has to be a way this can be fixed for 2.6.16?
I'll try it.
If it works, at least it might give us a clue to what to do.
-- Steve
On Fri, 2006-01-13 at 11:32 -0800, [email protected] wrote:
> On Fri, Jan 13, 2006 at 10:56:13AM -0800, Sven-Thorsten Dietrich wrote:
> > On Fri, 2006-01-13 at 10:55 -0800, [email protected] wrote:
> > > unless we can re-sync the TSCs often enough that apps don't notice.
> >
> > You'd have to quantify that somehow, in terms of the max drift rate
> > (ppm), and the max resolution available (< tsc frequency).
> >
> > Either that, or track an offset, and use one TSC as truth, and update
> > the correction factor for the other TSCs as often as needed, maybe?
> >
> > This is kind of analogous to the "drift" NTP calculates against a
> > free-running oscillator.
> >
> > So you'd be pushing that functionality deeper into the OS-core.
> >
> > Dave Mills had that "hardpps" stuff in there for a while, it might be a
> > starting point.
> >
> > Just some thoughts for now...
>
> There's some chatter here about a solution involving a lazy sync of TSCs
> to the HPET (or other) whenever an app calls rdtsc after a potentially
> unsyncing event.
>
> For example, 'hlt' will initiate C1 which may cause clock ramping (and TSC
> skew). We can trap rdtsc after a hlt and re-sync the TSCs to some truly
> monotonic source, like HPET.
>
> I don't have all the details, some problems remain, and the work is not
> quite done yet, but it looks promising.
>
> Even if we eventually get synced TSCs, it's too little too late.
> basically, anything in-kernel that uses rdtsc is bound to break, and any
> app that uses rdtsc had better be using CPU affinity.
I know this might be somewhat of an overhead, but what about resyncing
on wakeup of the idle function? Isn't that where they fall apart?
-- Steve
On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote:
> On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote:
>
> > Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
> > unconditionally.
>
> Can you try
>
> http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch
>
> please ?
>
> tglx
>
>
Hmm, it doesn't compile :(
-- Steve
rostedt@gandalf:~/work/kernels/linux-2.6.15-hrt2$ make
amdmake
CHK include/linux/version.h
CC arch/x86_64/kernel/asm-offsets.s
In file included from include/linux/timex.h:61,
from include/linux/sched.h:11,
from arch/x86_64/kernel/asm-offsets.c:7:
include/asm/timex.h: In function 'get_cycles_sync':
include/asm/timex.h:36: warning: implicit declaration of function 'alternative_io'
include/asm/timex.h:37: error: called object '"=a"' is not a function
include/asm/timex.h:37: error: called object '"0"' is not a function
include/asm/timex.h:37: error: syntax error before ':' token
make[1]: *** [arch/x86_64/kernel/asm-offsets.s] Error 1
make: *** [prepare0] Error 2
On Fri, 2006-01-13 at 16:05 -0500, Steven Rostedt wrote:
> On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote:
> > On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote:
> >
> > > Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
> > > unconditionally.
> >
> > Can you try
> >
> > http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch
> >
> > please ?
> >
> > tglx
> >
> >
>
> Hmm, it doesn't compile :(
Grmbl, I check this tomorrow. Getting late here
tglx
On Fri, 13 Jan 2006, Thomas Gleixner wrote:
> On Fri, 2006-01-13 at 16:05 -0500, Steven Rostedt wrote:
> > On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote:
> > > On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote:
> > >
> > > > Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
> > > > unconditionally.
> > >
> > > Can you try
> > >
> > > http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch
> > >
> > > please ?
> > >
> > > tglx
> > >
> > >
> >
> > Hmm, it doesn't compile :(
>
> Grmbl, I check this tomorrow. Getting late here
>
OK, I'll probably wont work on it till Monday then (wifes rules ;)
-- Steve
On Fri, 13 Jan 2006, Lee Revell wrote:
> On Fri, 2006-01-13 at 10:56 -0800, Sven-Thorsten Dietrich wrote:
>> On Fri, 2006-01-13 at 10:55 -0800, [email protected] wrote:
>>> On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote:
>>>>> Some apps/users need higher resolution and lower overhead that only rdtsc
>>>>> can offer currently.
>>>>
>>>> But obviously if the TSC gives wildly inaccurate results, it cannot be
>>>> used no matter how low the overhead.
>>>
>>> unless we can re-sync the TSCs often enough that apps don't notice.
>>>
>>
>> You'd have to quantify that somehow, in terms of the max drift rate
>> (ppm), and the max resolution available (< tsc frequency).
>>
>> Either that, or track an offset, and use one TSC as truth, and update
>> the correction factor for the other TSCs as often as needed, maybe?
>>
>> This is kind of analogous to the "drift" NTP calculates against a
>> free-running oscillator.
>>
>> So you'd be pushing that functionality deeper into the OS-core.
>>
>> Dave Mills had that "hardpps" stuff in there for a while, it might be a
>> starting point.
>>
>> Just some thoughts for now...
>>
>
> It kind of makes you wonder what in the heck AMD were thinking, whether
> they realized that this design decision would cause so many problems at
> the OS level (it's broken at least Linux and Solaris). Maybe Windows
> keeps time in a way that was unaffected by this?
Lee, the last time I saw this discussion I thought it was identified that
the multiple cores (or IIRC the multiple chips in a SMP motherboard) would
only get out of sync when power management calls were made (hlt or
switching the c-state). IIRC the workaround that was posted then was to
just disable these in the kernel build.
if this is the case then I could easily see the hardware engineers
thinking that if the software did a sleep thing then it was up to the
software to re-sync the TSC clocks when the wakeup takes place.
David Lang
--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare
On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote:
> Lee, the last time I saw this discussion I thought it was identified that
> the multiple cores (or IIRC the multiple chips in a SMP motherboard) would
> only get out of sync when power management calls were made (hlt or
> switching the c-state). IIRC the workaround that was posted then was to
> just disable these in the kernel build.
not using 'hlt' when idling means that you spend 10s of Watts more power
on mostly idle systems.
On Fri, 13 Jan 2006 [email protected] wrote:
> On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote:
>> Lee, the last time I saw this discussion I thought it was identified that
>> the multiple cores (or IIRC the multiple chips in a SMP motherboard) would
>> only get out of sync when power management calls were made (hlt or
>> switching the c-state). IIRC the workaround that was posted then was to
>> just disable these in the kernel build.
>
> not using 'hlt' when idling means that you spend 10s of Watts more power
> on mostly idle systems.
true, but for people who need better time accruacy then the other
workaround this may be very acceptable.
David Lang
--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare
On Fri, Jan 13, 2006 at 02:05:23PM -0800, David Lang wrote:
> On Fri, 13 Jan 2006 [email protected] wrote:
>
> >On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote:
> >>Lee, the last time I saw this discussion I thought it was identified that
> >>the multiple cores (or IIRC the multiple chips in a SMP motherboard) would
> >>only get out of sync when power management calls were made (hlt or
> >>switching the c-state). IIRC the workaround that was posted then was to
> >>just disable these in the kernel build.
> >
> >not using 'hlt' when idling means that you spend 10s of Watts more power
> >on mostly idle systems.
>
> true, but for people who need better time accruacy then the other
> workaround this may be very acceptable.
sure, if power doesn't matter use idle=poll => problem solved.
On Fri, 2006-01-13 at 14:05 -0800, David Lang wrote:
> On Fri, 13 Jan 2006 [email protected] wrote:
>
> > On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote:
> >> Lee, the last time I saw this discussion I thought it was identified that
> >> the multiple cores (or IIRC the multiple chips in a SMP motherboard) would
> >> only get out of sync when power management calls were made (hlt or
> >> switching the c-state). IIRC the workaround that was posted then was to
> >> just disable these in the kernel build.
> >
> > not using 'hlt' when idling means that you spend 10s of Watts more power
> > on mostly idle systems.
>
> true, but for people who need better time accruacy then the other
> workaround this may be very acceptable.
>
1/4 KW / day for time synchronisation.
The power company would love that.
> David Lang
>
--
***********************************
Sven-Thorsten Dietrich
Real-Time Software Architect
MontaVista Software, Inc.
1237 East Arques Ave.
Sunnyvale, CA 94085
Phone: 408.992.4515
Fax: 408.328.9204
http://www.mvista.com
Platform To Innovate
***********************************
On Fri, 13 Jan 2006, Sven-Thorsten Dietrich wrote:
> On Fri, 2006-01-13 at 14:05 -0800, David Lang wrote:
>> On Fri, 13 Jan 2006 [email protected] wrote:
>>
>>> On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote:
>>>> Lee, the last time I saw this discussion I thought it was identified that
>>>> the multiple cores (or IIRC the multiple chips in a SMP motherboard) would
>>>> only get out of sync when power management calls were made (hlt or
>>>> switching the c-state). IIRC the workaround that was posted then was to
>>>> just disable these in the kernel build.
>>>
>>> not using 'hlt' when idling means that you spend 10s of Watts more power
>>> on mostly idle systems.
>>
>> true, but for people who need better time accruacy then the other
>> workaround this may be very acceptable.
>>
>
> 1/4 KW / day for time synchronisation.
>
> The power company would love that.
more precisely 1/4 KW Hour / day
$0.01 - $0.02/day (I had to lookup the current rates)
they probably won't notice.
David Lang
--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare
On Fri, 2006-01-13 at 15:09 -0500, Steven Rostedt wrote:
> On Fri, 2006-01-13 at 13:55 -0500, Lee Revell wrote:
>
> > >
> > > If that's what you want to know?
> >
> > I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually
> > guaranteed to be monotonic on these machines AKA do we break POSIX
> > compliance or not.
>
> Nope it doesn't work :-(
>
> I ran this test:
> http://www.kihontech.com/tests/rt/monotonic.c
>
> And got this:
>
> $ ./monotonic
> main program pid=8477
> hello from thread 0!
> hello from thread 1!
> hello from thread 2!
> hello from thread 3!
> hello from thread 4!
> Failed! prev: 6279.925610302 current: 6279.874615349
Not sure, but I think your test is broken.
While it serializes the checking and exchange of current and last, it
doesn't serialize the actual calling of clock_gettime().
Thus you can get something like:
last=0
A: B:
1: now = clock_gettime()
2: now = clock_gettime()
3: atomic {
test(now, last) [2>0]
last = now
}
4: atomic {
test(now, last) [1>2]
FAIL!
}
Attached is a similar testcase that I've been using myself that avoids
this issue (although I just converted it from gettimeofday to
clock_gettime, so there may still be an issue). Let me know if you have
any comments on it.
thanks
-john
Lee Revell wrote:
>>>>But obviously if the TSC gives wildly inaccurate results, it cannot be
>>>>used no matter how low the overhead.
>>>
>>>unless we can re-sync the TSCs often enough that apps don't notice.
>>>
>>
>>You'd have to quantify that somehow, in terms of the max drift rate
>>(ppm), and the max resolution available (< tsc frequency).
>>
>>Either that, or track an offset, and use one TSC as truth, and update
>>the correction factor for the other TSCs as often as needed, maybe?
>>
>>This is kind of analogous to the "drift" NTP calculates against a
>>free-running oscillator.
>>
>>So you'd be pushing that functionality deeper into the OS-core.
>>
>>Dave Mills had that "hardpps" stuff in there for a while, it might be a
>>starting point.
>>
>>Just some thoughts for now...
>
> It kind of makes you wonder what in the heck AMD were thinking, whether
> they realized that this design decision would cause so many problems at
> the OS level (it's broken at least Linux and Solaris). Maybe Windows
> keeps time in a way that was unaffected by this?
Sounds to me like they are doing something like what was being mentioned
above:
http://support.microsoft.com/kb/896256/en-us
"When TSC does not increment monotonically, system components that use
the kernel KeQueryPerformanceCounter function may not work correctly. To
address this problem, Microsoft makes it possible for the ACPI Power
Management Timer to be used as the operating system timer that supports
the kernel KeQueryPerformanceCounter function. However, some programs
may directly access the TSC by bypassing the Windows timer APIs. The
multiple-processor Hardware Abstraction Layer (HAL) makes sure that the
TSC registers on all processors on a multiple-processor computer remain
closely synchronized. Therefore, access by system software that may be
directed to different processors does not return different results."
Also, Microsoft's docs for QueryPerformanceCounter specify that
different results on different CPUs will only occur if there are "bugs
in the basic input/output system (BIOS) or the hardware abstraction
layer (HAL)" and recommends that threads using this function set their
affinity to run on one processor only.
--
Robert Hancock Saskatoon, SK, Canada
To email, remove "nospam" from [email protected]
Home Page: http://www.roberthancock.com/
David Lang wrote:
> On Fri, 13 Jan 2006, Sven-Thorsten Dietrich wrote:
>
>> On Fri, 2006-01-13 at 14:05 -0800, David Lang wrote:
>>
>>> On Fri, 13 Jan 2006 [email protected] wrote:
>>>
>>>> On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote:
>>>>
>>>>> Lee, the last time I saw this discussion I thought it was
>>>>> identified that
>>>>> the multiple cores (or IIRC the multiple chips in a SMP
>>>>> motherboard) would
>>>>> only get out of sync when power management calls were made (hlt or
>>>>> switching the c-state). IIRC the workaround that was posted then
>>>>> was to
>>>>> just disable these in the kernel build.
>>>>
>>>>
>>>> not using 'hlt' when idling means that you spend 10s of Watts more
>>>> power
>>>> on mostly idle systems.
>>>
>>>
>>> true, but for people who need better time accruacy then the other
>>> workaround this may be very acceptable.
>>>
>>
>> 1/4 KW / day for time synchronisation.
>>
>> The power company would love that.
>
>
> more precisely 1/4 KW Hour / day
>
> $0.01 - $0.02/day (I had to lookup the current rates)
>
> they probably won't notice.
>
Well, wait until there's AMD based dual core x86_64 laptops out there
(this email being written on a single core x86_64 one). I can already
see the faces of the unhappy future owners being told "use idle=poll"
when on battery and anyway going deaf by fan noise.
(/me ducks and runs)
--
Andreas Steinmetz SPAMmers use [email protected]
On Sat, Jan 14, 2006 at 01:41:43AM +0100, Andreas Steinmetz wrote:
> > more precisely 1/4 KW Hour / day
> >
> > $0.01 - $0.02/day (I had to lookup the current rates)
> >
> > they probably won't notice.
>
> Well, wait until there's AMD based dual core x86_64 laptops out there
> (this email being written on a single core x86_64 one). I can already
> see the faces of the unhappy future owners being told "use idle=poll"
> when on battery and anyway going deaf by fan noise.
Yeah, and then multiply that cost by a few tens of thousands of servers.
idle=poll is a hack at best.
On Sat, 14 Jan 2006, Andreas Steinmetz wrote:
> David Lang wrote:
>> On Fri, 13 Jan 2006, Sven-Thorsten Dietrich wrote:
>>
>>> On Fri, 2006-01-13 at 14:05 -0800, David Lang wrote:
>>>
>>>> On Fri, 13 Jan 2006 [email protected] wrote:
>>>>
>>>>> On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote:
>>>>>
>>>>>> Lee, the last time I saw this discussion I thought it was
>>>>>> identified that
>>>>>> the multiple cores (or IIRC the multiple chips in a SMP
>>>>>> motherboard) would
>>>>>> only get out of sync when power management calls were made (hlt or
>>>>>> switching the c-state). IIRC the workaround that was posted then
>>>>>> was to
>>>>>> just disable these in the kernel build.
>>>>>
>>>>>
>>>>> not using 'hlt' when idling means that you spend 10s of Watts more
>>>>> power
>>>>> on mostly idle systems.
>>>>
>>>>
>>>> true, but for people who need better time accruacy then the other
>>>> workaround this may be very acceptable.
>>>>
>>>
>>> 1/4 KW / day for time synchronisation.
>>>
>>> The power company would love that.
>>
>>
>> more precisely 1/4 KW Hour / day
>>
>> $0.01 - $0.02/day (I had to lookup the current rates)
>>
>> they probably won't notice.
>>
>
> Well, wait until there's AMD based dual core x86_64 laptops out there
> (this email being written on a single core x86_64 one). I can already
> see the faces of the unhappy future owners being told "use idle=poll"
> when on battery and anyway going deaf by fan noise.
>
> (/me ducks and runs)
I'm not saying it's the right answer, but it's one of two workarounds
currently available.
idle=poll causes increased power useage
timer source change (mentioned earlier in this thread) limits timer
precision
neither of these are fixes, but by understanding the different costs
people can choose the work around they want to use while waiting for a
better fix.
David Lang
--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare
On Fri, 2006-01-13 at 16:23 -0800, john stultz wrote:
> > Failed! prev: 6279.925610302 current: 6279.874615349
>
> Not sure, but I think your test is broken.
Yep and it is now fixed:
http://www.kihontech.com/tests/rt/monotonic.c
>
> Attached is a similar testcase that I've been using myself that avoids
> this issue (although I just converted it from gettimeofday to
> clock_gettime, so there may still be an issue). Let me know if you have
> any comments on it.
Thanks, I'll add that to my list of tests too.
Oh and 2.6.15 passed as well (with clock=pmtmr)
-- Steve
On Fri, 2006-01-13 at 20:10 -0500, Steven Rostedt wrote:
>
> Thanks, I'll add that to my list of tests too.
>
> Oh and 2.6.15 passed as well (with clock=pmtmr)
It really seems like it would fail if you gave it enough time due to the
rdtsc in monotonic_clock()...
Lee
David Lang wrote:
[snip]
> I'm not saying it's the right answer, but it's one of two workarounds
> currently available.
>
> idle=poll causes increased power useage
>
> timer source change (mentioned earlier in this thread) limits timer
> precision
>
> neither of these are fixes, but by understanding the different costs
> people can choose the work around they want to use while waiting for a
> better fix.
The problem being that some of us use their laptops for audio work too.
And then high battery usage, noisy fans or lack of high res timers will
be really bad.
Simple example:
I do final mastering work using my laptop and Ardour/Jack/JAMin out of
house in a place with a good stage audio system (Mackie mixer, 2KW
Dynacord Amp/Syrincs S3) and a sufficient listening space to get a
proper bass mix. I run on battery in this case to avoid any kind of
audio interference (ground loops, etc). Now thinking of a dual core
laptop...
--
Andreas Steinmetz SPAMmers use [email protected]
On Fri, 2006-01-13 at 20:14 -0500, Lee Revell wrote:
> On Fri, 2006-01-13 at 20:10 -0500, Steven Rostedt wrote:
> >
> > Thanks, I'll add that to my list of tests too.
> >
> > Oh and 2.6.15 passed as well (with clock=pmtmr)
>
> It really seems like it would fail if you gave it enough time due to the
> rdtsc in monotonic_clock()...
Currently monotonic_clock()is only used by the hangcheck-timer, and is
not used by gettimeofday/clock_gettime (even w/ CLOCK_MONOTONIC).
So there may still be an issue there w/ the hangcheck-timer(for x86-64,
on i386 the acpi pm timer can be used for monotonic_clock), but its
doesn't affect the time related userland interfaces.
thanks
-john
On Fri, 2006-01-13 at 17:23 -0800, john stultz wrote:
> On Fri, 2006-01-13 at 20:14 -0500, Lee Revell wrote:
> > On Fri, 2006-01-13 at 20:10 -0500, Steven Rostedt wrote:
> > >
> > > Thanks, I'll add that to my list of tests too.
> > >
> > > Oh and 2.6.15 passed as well (with clock=pmtmr)
> >
> > It really seems like it would fail if you gave it enough time due to the
> > rdtsc in monotonic_clock()...
>
> Currently monotonic_clock()is only used by the hangcheck-timer, and is
> not used by gettimeofday/clock_gettime (even w/ CLOCK_MONOTONIC).
>
> So there may still be an issue there w/ the hangcheck-timer(for x86-64,
> on i386 the acpi pm timer can be used for monotonic_clock), but its
> doesn't affect the time related userland interfaces.
OK so the last question is how do we make sure the kernel uses the
clock=pmtmr behavior by default on those machines?
Lee
On Fri, 2006-01-13 at 20:27 -0500, Lee Revell wrote:
> On Fri, 2006-01-13 at 17:23 -0800, john stultz wrote:
> > On Fri, 2006-01-13 at 20:14 -0500, Lee Revell wrote:
> > > On Fri, 2006-01-13 at 20:10 -0500, Steven Rostedt wrote:
> > > >
> > > > Thanks, I'll add that to my list of tests too.
> > > >
> > > > Oh and 2.6.15 passed as well (with clock=pmtmr)
> > >
> > > It really seems like it would fail if you gave it enough time due to the
> > > rdtsc in monotonic_clock()...
> >
> > Currently monotonic_clock()is only used by the hangcheck-timer, and is
> > not used by gettimeofday/clock_gettime (even w/ CLOCK_MONOTONIC).
> >
> > So there may still be an issue there w/ the hangcheck-timer(for x86-64,
> > on i386 the acpi pm timer can be used for monotonic_clock), but its
> > doesn't affect the time related userland interfaces.
>
> OK so the last question is how do we make sure the kernel uses the
> clock=pmtmr behavior by default on those machines?
This is as I understand it:
With 2.6.15 on x86-64:
If available, alternate timesources (HPET, ACPI PM) will be used if
available on AMD SMP systems. (clock= is i386 only)
With 2.6.15 on i386:
If CONFIG_X86_PM_TIMER is enabled, and available it is the preferred
clocksource over the TSC. Some distros have changed this priority
causing the TSC to be preferred. In these cases clock=pmtmr is needed.
How's that?
-john
On Sat, Jan 14, 2006 at 02:21:23AM +0100, Andreas Steinmetz wrote:
> > neither of these are fixes, but by understanding the different costs
> > people can choose the work around they want to use while waiting for a
> > better fix.
>
> The problem being that some of us use their laptops for audio work too.
> And then high battery usage, noisy fans or lack of high res timers will
> be really bad.
>
> Simple example:
> I do final mastering work using my laptop and Ardour/Jack/JAMin out of
> house in a place with a good stage audio system (Mackie mixer, 2KW
> Dynacord Amp/Syrincs S3) and a sufficient listening space to get a
> proper bass mix. I run on battery in this case to avoid any kind of
> audio interference (ground loops, etc). Now thinking of a dual core
> laptop...
Fixes are coming, just need to sort out the fidlly bits.
On Fri, 13 Jan 2006, john stultz wrote:
>
> This is as I understand it:
>
> With 2.6.15 on x86-64:
> If available, alternate timesources (HPET, ACPI PM) will be used if
> available on AMD SMP systems. (clock= is i386 only)
Hmm, should I boot without the clock= to prove this?
-- Steve
>
> With 2.6.15 on i386:
> If CONFIG_X86_PM_TIMER is enabled, and available it is the preferred
> clocksource over the TSC. Some distros have changed this priority
> causing the TSC to be preferred. In these cases clock=pmtmr is needed.
>
> How's that?
> -john
>
>
>
On Fri, 2006-01-13 at 20:43 -0500, Steven Rostedt wrote:
> On Fri, 13 Jan 2006, john stultz wrote:
>
> >
> > This is as I understand it:
> >
> > With 2.6.15 on x86-64:
> > If available, alternate timesources (HPET, ACPI PM) will be used if
> > available on AMD SMP systems. (clock= is i386 only)
>
> Hmm, should I boot without the clock= to prove this?
Feel free. That or grep the x86-64 time.c code.
Look for:
time.c: Using PM based timekeeping.
To verify the timesource selection.
thanks
-john
On Fri, 13 Jan 2006, john stultz wrote:
> On Fri, 2006-01-13 at 20:43 -0500, Steven Rostedt wrote:
> > On Fri, 13 Jan 2006, john stultz wrote:
> >
> > >
> > > This is as I understand it:
> > >
> > > With 2.6.15 on x86-64:
> > > If available, alternate timesources (HPET, ACPI PM) will be used if
> > > available on AMD SMP systems. (clock= is i386 only)
> >
> > Hmm, should I boot without the clock= to prove this?
>
> Feel free. That or grep the x86-64 time.c code.
>
> Look for:
> time.c: Using PM based timekeeping.
>
> To verify the timesource selection.
I'm too busy trying to install linux on my powerbook to look at code at
the moment :-)
I just rebooted and ran the test on 2.6.15 without the clock= option and
it passed my test. I'll wait a while and run it again to see if I get any
other drift too prove it more.
-- Steve
On Fri, 2006-01-13 at 17:50 -0800, john stultz wrote:
> On Fri, 2006-01-13 at 20:43 -0500, Steven Rostedt wrote:
> > On Fri, 13 Jan 2006, john stultz wrote:
> >
> > >
> > > This is as I understand it:
> > >
> > > With 2.6.15 on x86-64:
> > > If available, alternate timesources (HPET, ACPI PM) will be used if
> > > available on AMD SMP systems. (clock= is i386 only)
> >
> > Hmm, should I boot without the clock= to prove this?
>
> Feel free. That or grep the x86-64 time.c code.
>
> Look for:
> time.c: Using PM based timekeeping.
>
> To verify the timesource selection.
So much for not looking at code!
I uploaded yet another version of
http://www.kihontech.com/tests/rt/monotonic.c
that now also tests the tsc too, to see if it goes backwards. If it
does, then it prints (only once) that fact and continues testing the
gettime function.
So this proves that the tsc does go back and the gettime still is
working.
$ ./monotonic
main program pid=7137
hello from thread 0!
hello from thread 4!
last tsc is 4389431102233 this tsc is 4389429636684
hello from thread 1!
hello from thread 2!
hello from thread 3!
-- Steve
On Fri, 13 Jan 2006, Lee Revell wrote:
> I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually
> guaranteed to be monotonic on these machines AKA do we break POSIX
> compliance or not.
Yes you do. Other arches have had the same issues.... See IA64 code.
On Fri, 2006-01-13 at 17:04 -0800, David Lang wrote:
> On Sat, 14 Jan 2006, Andreas Steinmetz wrote:
>
> > David Lang wrote:
> > Well, wait until there's AMD based dual core x86_64 laptops out there
> > (this email being written on a single core x86_64 one). I can already
> > see the faces of the unhappy future owners being told "use idle=poll"
> > when on battery and anyway going deaf by fan noise.
> >
> > (/me ducks and runs)
>
> I'm not saying it's the right answer, but it's one of two workarounds
> currently available.
>
> idle=poll causes increased power useage
>
> timer source change (mentioned earlier in this thread) limits timer
> precision
>
> neither of these are fixes, but by understanding the different costs
> people can choose the work around they want to use while waiting for a
> better fix.
A laptop user could also bind a process to a single CPU, and use the
scaling min/max values to lock CPU speed to a single value. The TSC may
still stop during HLT, but software must be handling that already.
Wouldn't that provide an accurate TSC?
--
Zan Lynx <[email protected]>
On Sun, Jan 15, 2006 at 01:52:44AM -0700, Zan Lynx wrote:
> A laptop user could also bind a process to a single CPU, and use the
> scaling min/max values to lock CPU speed to a single value. The TSC may
> still stop during HLT, but software must be handling that already.
>
> Wouldn't that provide an accurate TSC?
monotonic but not linear. Also remember that the OS will use rdtsc here
and there, and you can't affine the OS :)
On Sun, 2006-01-15 at 08:25 -0800, [email protected] wrote:
> On Sun, Jan 15, 2006 at 01:52:44AM -0700, Zan Lynx wrote:
> > A laptop user could also bind a process to a single CPU, and use the
> > scaling min/max values to lock CPU speed to a single value. The TSC may
> > still stop during HLT, but software must be handling that already.
> >
> > Wouldn't that provide an accurate TSC?
>
> monotonic but not linear. Also remember that the OS will use rdtsc here
> and there, and you can't affine the OS :)
>
So the options are either to fix the TSC handling on these systems (by
resyncing the TSCs when exiting from HLT), or eliminate the use of rdtsc
by the OS?
Lee
On Sun, Jan 15, 2006 at 11:33:36AM -0500, Lee Revell wrote:
> On Sun, 2006-01-15 at 08:25 -0800, [email protected] wrote:
> > On Sun, Jan 15, 2006 at 01:52:44AM -0700, Zan Lynx wrote:
> > > A laptop user could also bind a process to a single CPU, and use the
> > > scaling min/max values to lock CPU speed to a single value. The TSC may
> > > still stop during HLT, but software must be handling that already.
> > >
> > > Wouldn't that provide an accurate TSC?
> >
> > monotonic but not linear. Also remember that the OS will use rdtsc here
> > and there, and you can't affine the OS :)
> >
>
> So the options are either to fix the TSC handling on these systems (by
> resyncing the TSCs when exiting from HLT), or eliminate the use of rdtsc
> by the OS?
Or both. You can trap rdtsc users in userland, you have to manually audit
kernel users.
It should be abolished or properly wrapped in kernel code, as soon as a
resync path is available.
On Sun, 2006-01-15 at 10:21 -0800, [email protected] wrote:
> On Sun, Jan 15, 2006 at 11:33:36AM -0500, Lee Revell wrote:
> > On Sun, 2006-01-15 at 08:25 -0800, [email protected] wrote:
> > > On Sun, Jan 15, 2006 at 01:52:44AM -0700, Zan Lynx wrote:
> > > > A laptop user could also bind a process to a single CPU, and use the
> > > > scaling min/max values to lock CPU speed to a single value. The TSC may
> > > > still stop during HLT, but software must be handling that already.
> > > >
> > > > Wouldn't that provide an accurate TSC?
> > >
> > > monotonic but not linear. Also remember that the OS will use rdtsc here
> > > and there, and you can't affine the OS :)
> > >
> >
> > So the options are either to fix the TSC handling on these systems (by
> > resyncing the TSCs when exiting from HLT), or eliminate the use of rdtsc
> > by the OS?
>
> Or both. You can trap rdtsc users in userland, you have to manually audit
> kernel users.
>
> It should be abolished or properly wrapped in kernel code, as soon as a
> resync path is available.
>
For the purposes of this discussion I was not considering direct use of
rdtsc from userland for timing, I've accepted the arguments that this is
a lost cause with today's hardware (although I maintain it was viable in
practice on previous generations of hardware). Besides, rdtsc is
useless from userspace if the kernel traps it, because the whole point
of using it over gettimeofday() was that it was dirt cheap.
I'm content to make sure the kernel's internal timekeeping is solid.
Lee
On Sun, Jan 15, 2006 at 01:29:36PM -0500, Lee Revell wrote:
> > Or both. You can trap rdtsc users in userland, you have to manually audit
> > kernel users.
> >
> > It should be abolished or properly wrapped in kernel code, as soon as a
> > resync path is available.
>
> For the purposes of this discussion I was not considering direct use of
> rdtsc from userland for timing, I've accepted the arguments that this is
> a lost cause with today's hardware (although I maintain it was viable in
> practice on previous generations of hardware). Besides, rdtsc is
> useless from userspace if the kernel traps it, because the whole point
> of using it over gettimeofday() was that it was dirt cheap.
>
> I'm content to make sure the kernel's internal timekeeping is solid.
You only need to trap rdtsc if you know that some unsyncronizing event has
happened. E.g. 'hlt' or some other sleep state. If we sync every time we
hlt, then we don't need to trap. But we can maybe be lazier about it.
The sync operation itself may not be cheap, and it may be useful to skip
it if possible.
john stultz <[email protected]> writes:
>
> With 2.6.15 on x86-64:
> If available, alternate timesources (HPET, ACPI PM) will be used if
> available on AMD SMP systems. (clock= is i386 only)
It would be good if it worked on x86-64 too - simply to unconfuse people.
It's somewhere on my todo list, but patches welcome.
> With 2.6.15 on i386:
> If CONFIG_X86_PM_TIMER is enabled, and available it is the preferred
> clocksource over the TSC. Some distros have changed this priority
> causing the TSC to be preferred. In these cases clock=pmtmr is needed.
One problem is that it is not obvious enough to people that
CONFIG_X86_PM_TIMER is really needed for correct timing on many system
and they just don't enable it. That is why we got so many bogus reports.
I just changed it on x86-64 to be dependent on EMBEDDED, on by
default. i386 probably should do this change too.
-Andi
On Fri, 2006-01-13 at 16:17 -0500, Steven Rostedt wrote:
> > >
> > > Hmm, it doesn't compile :(
> >
> > Grmbl, I check this tomorrow. Getting late here
> >
>
> OK, I'll probably wont work on it till Monday then (wifes rules ;)
Can you try this one please ?
http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt3.patch
tglx
On Mon, 2006-01-16 at 10:52 +0100, Thomas Gleixner wrote:
> On Fri, 2006-01-13 at 16:17 -0500, Steven Rostedt wrote:
> > > >
> > > > Hmm, it doesn't compile :(
> > >
> > > Grmbl, I check this tomorrow. Getting late here
> > >
> >
> > OK, I'll probably wont work on it till Monday then (wifes rules ;)
>
> Can you try this one please ?
>
> http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt3.patch
It compiled.
I'll boot into it now.
-- Steve