Posix compliant cpu clocks: V7 [0/3] Rationale and test program
Signed-off-by: Christoph Lameter <[email protected]>
Changes from V5:
* add a simple means of accessing other processes cputimers.
* Simplified glibc patch that makes glibc return errors when
an applications accesses cputimers and is run with glibc under a kernel
not providing cputimers. This may allow the detection of applications that
need to be updated since they depend on the earlier non posix compliant
version of glibc. Also should be more readable since the sections
are no longer moved betweeen directory hierachies.
* remove ability to set process and thread clocks.
POSIX clocks are to be implemented in the following way according
to V3 of the Single Unix Specification:
1. CLOCK_PROCESS_CPUTIME_ID
Implementations shall also support the special clockid_t value
CLOCK_PROCESS_CPUTIME_ID, which represents the CPU-time clock of the
calling process when invoking one of the clock_*() or timer_*()
functions. For these clock IDs, the values returned by clock_gettime() and
specified by clock_settime() represent the amount of execution time of the
process associated with the clock.
2. CLOCK_THREAD_CPUTIME_ID
Implementations shall also support the special clockid_t value
CLOCK_THREAD_CPUTIME_ID, which represents the CPU-time clock of the
calling thread when invoking one of the clock_*() or timer_*()
functions. For these clock IDs, the values returned by clock_gettime()
and specified by clock_settime() shall represent the amount of
execution time of the thread associated with the clock.
These times mentioned are CPU processing times and not the time that has
passed since the startup of a process. Glibc currently provides its own
implementation of these two clocks which is designed to return the time
that passed since the startup of a process or a thread.
Moreover Glibc's clocks are bound to CPU timers which is problematic when the
frequency of the clock changes or the process is moved to a different
processor whose cpu timer may not be fully synchronized to the cpu timer
of the current CPU. This patchset results in a both clocks working reliably.
The patch also implements the access to other the thread and process clocks
of linux processes by using negative clockid's:
1. For CLOCK_PROCESS_CPUTIME_ID: -pid
2. For CLOCK_THREAD_CPUTIME_ID: -(pid + PID_MAX_LIMIT)
This allows
clock_getcpuclockid(pid) to return -pid
and
pthread_getcpuiclock(pid) to return -(pid + PID_MAX_LIMIT)
to allow access to the corresponding clocks.
Todo:
- The timer API to generate events by a non tick based timer is not
usable in its current state. The posix timer API seems to be only
useful at this point to define clock_get/set. Need to revise this.
- Implement timed interrupts in mmtimer after API is revised.
The patchset consists of the following components:
[0/2] Contains an explanation as to why these patches are necessary
as well as a test program and the output of a sample run.
[1/2] Linux Kernel Patch: Implements the two clocks and enhances some
pieces of the posix-timers implementation in the kernel for these
clocks and also makes it possible for device drivers to define
additional clocks.
[2/2] Glibc patch: Use kernel clocks and also makes glibc able to use any
posix clock provided by the kernel so that posix clocks by driver
may be accessed. Break apps that use the old cputimer based
CLOCK_*_CPUTIME_IDs on kernels that do not support CLOCK_*_CPUTIME_ID.
The mmtimer patch is unchanged from V6 and stays as is in 2.6.9-rc3-mm2.
But I expect to update the driver as soon as the interface to setup hardware
timer interrupts is usable.
Rolands patches:
----------------
This patch is not as comprehensive as Rolands but also not as invasive.
Roland's high resolution time through tsc is a hack but would significantly
improve the statistics we get for a process. However, we may get the same
result in a cleaner way after John Stultz and my plan for revising the
time subsystem is through. This would replace jiffies with nanoseconds
for time keeping throughout the kernel.
Nanosecond accuracy is already available through getnstimeofday() on some
platforms. Some of the functionality in Roland's patch may be implemented in a
TSC independent way using this function.
The use of negative pid values in certain numeric ranges for selecting the
type of clock is something that is also in a simplerr way present in
my patch. I do not like this and would rather see this incorporated in
glibc if the glibc people can come up with a sane implementation.
Single Thread Testing
CLOCK_THREAD_CPUTIME_ID= 0.494140878 resolution= 0.000976563
CLOCK_PROCESS_CPUTIME_ID= 0.494140878 resolution= 0.000976563
Multi Thread Testing
Starting Thread: 0 1 2 3 4 5 6 7 8 9
Joining Thread: 0 1 2 3 4 5 6 7 8 9
0 Cycles= 0 Thread= 0.000000000ns Process= 0.495117441ns
1 Cycles=1000000 Thread= 0.140625072ns Process= 2.523438792ns
2 Cycles=2000000 Thread= 0.966797370ns Process= 8.512699671ns
3 Cycles=3000000 Thread= 0.806641038ns Process= 7.561527309ns
4 Cycles=4000000 Thread= 1.865235330ns Process= 12.891608163ns
5 Cycles=5000000 Thread= 1.604493009ns Process= 11.528326215ns
6 Cycles=6000000 Thread= 2.086915131ns Process= 13.500983475ns
7 Cycles=7000000 Thread= 2.245118337ns Process= 13.947272766ns
8 Cycles=8000000 Thread= 1.604493009ns Process= 12.252935961ns
9 Cycles=9000000 Thread= 2.160157356ns Process= 13.977546219ns
Clock status at the end of the timer tests:
Gettimeofday() = 1097084999.489938000
CLOCK_REALTIME= 1097084999.490116229 resolution= 0.000000040
CLOCK_MONOTONIC= 177.071675109 resolution= 0.000000040
CLOCK_PROCESS_CPUTIME_ID= 13.978522782 resolution= 0.000976563
CLOCK_THREAD_CPUTIME_ID= 0.497070567 resolution= 0.000976563
CLOCK_SGI_CYCLE= 229.967982280 resolution= 0.000000040
PROCESS clock of 1 (init)= 4.833986850 resolution= 0.000976563
THREAD clock of 1 (init)= 0.009765630 resolution= 0.000976563
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <errno.h>
#include <asm/unistd.h>
#include <pthread.h>
#define clock_getres(x,y) syscall(__NR_clock_getres, x,y)
#define clock_gettime(x,y) syscall(__NR_clock_gettime, x, y)
#define clock_settime(x,y) syscall(__NR_clock_settime, x, y)
void pr(int clock,const char *n)
{
struct timespec tv = {1,2};
struct timespec res = {3,4};
int rc;
rc=clock_getres(clock,&res);
if (rc) {
printf("getres return code on %s=%d errno=%d\n",n,rc,errno);
}
rc=clock_gettime(clock,&tv);
if (rc) {
printf("gettime return code on %s=%d errno=%d\n",n,rc, errno);
}
else
printf("%25s=% 11d.%09d resolution=% 2d.%09d\n",n,tv.tv_sec,tv.tv_nsec,res.tv_sec,res.tv_nsec);
}
int y;
void kx(long long x) {
y=x;
};
struct timespec zero;
pthread_t thread[10];
struct tinfo {
int i;
struct timespec ttime,ptime;
} tinf[10];
void *thread_function(void *x) {
struct tinfo *t=x;
int i;
for(i=1;i< t->i;i++) kx(1000000000000LL/i);
clock_gettime(CLOCK_THREAD_CPUTIME_ID,&t->ttime);
clock_gettime(CLOCK_PROCESS_CPUTIME_ID,&t->ptime);
}
int main(char argc, char *argv[])
{
struct timespec tv;
int i;
/* Waste some time */
printf("Single Thread Testing\n");
for(i=1;i<10000000;i++) kx(1000000000000LL/i);
pr(CLOCK_THREAD_CPUTIME_ID,"CLOCK_THREAD_CPUTIME_ID");
pr(CLOCK_PROCESS_CPUTIME_ID,"CLOCK_PROCESS_CPUTIME_ID");
/* Waste some more time in threads */
printf("Multi Thread Testing\nStarting Thread:");
clock_settime(CLOCK_PROCESS_CPUTIME_ID,&zero);
for(i=0;i<10;i++) {
tinf[i].i=i*1000000;
if (pthread_create(&thread[i], NULL, thread_function, tinf+i))
perror("thread");
else
printf(" %d",i);
}
printf("\n Joining Thread:");
for(i=0;i<10;i++) if (pthread_join( thread[i], NULL)) perror("join"); else printf(" %d",i);
printf("\n");
for(i=0;i<10;i++) {
printf("%d Cycles=%7d Thread=% 3d.%09dns Process=% 3d.%09dns\n",i,tinf[i].i,tinf[i].ttime.tv_sec,tinf[i].ttime.tv_nsec,tinf[i].ptime.tv_sec,tinf[i].ptime.tv_nsec);
}
gettimeofday((struct timeval *)&tv);
tv.tv_nsec = tv.tv_nsec*1000;
printf("\nClock status at the end of the timer tests:\n");
printf(" Gettimeofday() =% 11d.%09d\n",tv.tv_sec,tv.tv_nsec);
pr(CLOCK_REALTIME,"CLOCK_REALTIME");
pr(CLOCK_MONOTONIC,"CLOCK_MONOTONIC");
pr(CLOCK_PROCESS_CPUTIME_ID,"CLOCK_PROCESS_CPUTIME_ID");
pr(CLOCK_THREAD_CPUTIME_ID,"CLOCK_THREAD_CPUTIME_ID");
pr(10,"CLOCK_SGI_CYCLE");
pr(-1,"PROCESS clock of 1 (init)");
pr(-1 - 4*1024*1024,"THREAD clock of 1 (init)");
printf("\n");
}
I ran some test programs and discovered that the periodic timer support
is broken. The timer is triggered once and then never again. Single shot
timers work fine. 2.6.9-rc1 is fine. The first kernel that I tested where
I noticed the breakage was 2.6.9-rc1-bk17. 2.6.9-rc2 and following all
cannot do periodic timer signals.
I looked through the changelog but I cannot see anything that would cause
the problem. Roland's patch surely could not have done this.
Will try to track this down further, time permitting...
Christoph Lameter wrote:
> I ran some test programs and discovered that the periodic timer support
> is broken. The timer is triggered once and then never again. Single shot
> timers work fine. 2.6.9-rc1 is fine. The first kernel that I tested where
> I noticed the breakage was 2.6.9-rc1-bk17. 2.6.9-rc2 and following all
> cannot do periodic timer signals.
>
> I looked through the changelog but I cannot see anything that would cause
> the problem. Roland's patch surely could not have done this.
>
> Will try to track this down further, time permitting...
The most likely thing would be failure to do the call back from the signal
delivery code.
--
George Anzinger [email protected]
High-res-timers: http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
> I ran some test programs and discovered that the periodic timer support
> is broken. The timer is triggered once and then never again. Single shot
> timers work fine. 2.6.9-rc1 is fine. The first kernel that I tested where
> I noticed the breakage was 2.6.9-rc1-bk17. 2.6.9-rc2 and following all
> cannot do periodic timer signals.
>
> I looked through the changelog but I cannot see anything that would cause
> the problem. Roland's patch surely could not have done this.
>
> Will try to track this down further, time permitting...
I took a bit of a look at this, and it looks like some things changed
with the introduction of the flexible mmap in 2.6.9-rc1-bk1.
If you run the program below it will work, doing as expected. Now
comment out the the line "memset(&sa, 0, sizeof(struct sigaction));"
and program won't run as expected.
Now do "echo -n 1 > /proc/sys/vm/legacy_va_layout" and run the same
program again (the one with memset commented out).
Turning on signal debugging tells us that with legacy_va_layout=0
"SIG deliver (a.out:415): sp=bffff6c0 pc=08048434 ra=00000000"
where ra is the 8-byte instruction that's supposed to get us back to
sys_sigreturn().
Me thinks someone somewhere is using some of the bits that we
"accidently" pass via sa.sa_flags by not setting it to 0, the regular
flags don't seem to show this behaviour, and I couldn't see any real
checking of the passed value of sa.sa_flags.
---------------------------------------------------------
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/time.h>
void sighandler(int signal)
{
printf("hihi\n");
}
int main()
{
struct itimerval timeval;
struct sigaction sa;
memset(&timeval, 0, sizeof(struct timeval));
memset(&sa, 0, sizeof(struct sigaction));
sa.sa_handler = &sighandler;
sigfillset(&sa.sa_mask);
sigaction(SIGALRM, &sa, NULL);
timeval.it_interval.tv_sec = 2;
timeval.it_interval.tv_usec = 0;
timeval.it_value.tv_sec = 2;
timeval.it_value.tv_usec = 0;
if (setitimer(ITIMER_REAL, &timeval, NULL))
printf("Nooo!\n");
for(;;)
;
return 0;
}
> Me thinks someone somewhere is using some of the bits that we
> "accidently" pass via sa.sa_flags by not setting it to 0, the regular
> flags don't seem to show this behaviour, and I couldn't see any real
> checking of the passed value of sa.sa_flags.
Nope. It was my screwed up setting of the resolution of CLOCK_REALTIME and
CLOCK_MONOTONIC. Fix was submitted to Linus and Andrew.