Hello,
It is possible to ask Linux to timestamp incoming datagrams when they
are received, then to retrieve this timestamp with an ioctl command or
a recvmsg call (which would save one round trip to kernel space).
SIOCGSTAMP
Return a struct timeval with the receive timestamp of the last
packet passed to the user. This is useful for accurate round trip time
measurements. See setitimer(2) for a description of struct timeval.
As far as I understand, this timestamp is given by the CLOCK_REALTIME
clock. I would like to get the timestamp given by a different clock: the
CLOCK_MONOTONIC clock.
In other words, I would like the kernel to do the equivalent of
struct timespec spec;
clock_gettime(CLOCK_MONOTONIC, &spec)
for each datagram the system receives, as soon as it is received.
Is there a way to achieve that?
Is there a different ioctl perhaps? (I don't think so.)
Regards.
John wrote:
> It is possible to ask Linux to timestamp incoming datagrams when they
> are received, then to retrieve this timestamp with an ioctl command or
> a recvmsg call (which would save one round trip to kernel space).
>
> SIOCGSTAMP
> Return a struct timeval with the receive timestamp of the last
> packet passed to the user. This is useful for accurate round trip time
> measurements. See setitimer(2) for a description of struct timeval.
>
> As far as I understand, this timestamp is given by the CLOCK_REALTIME
> clock. I would like to get the timestamp given by a different clock:
> the CLOCK_MONOTONIC clock.
>
> In other words, I would like the kernel to do the equivalent of
>
> struct timespec spec;
> clock_gettime(CLOCK_MONOTONIC, &spec)
>
> for each datagram the system receives, as soon as it is received.
>
> Is there a way to achieve that?
>
> Is there a different ioctl perhaps? (I don't think so.)
I've been reading the following documents:
Clocks and Timers
http://www.opengroup.org/onlinepubs/009695399/xrat/xsh_chap02.html#tag_03_02_08_18
Rationale for the Monotonic Clock
http://www.opengroup.org/onlinepubs/009695399/xrat/xsh_chap02.html#tag_03_02_08_19
"For those applications that use time services to achieve realtime
behavior, changing the value of the clock on which these services rely
may cause erroneous timing behavior. For these applications, it is
necessary to have a monotonic clock which cannot run backwards, and
which has a maximum clock jump that is required to be documented by the
implementation. Additionally, it is desirable (but not required by IEEE
Std 1003.1-2001) that the monotonic clock increases its value uniformly.
This clock should not be affected by changes to the system time; for
example, to synchronize the clock with an external source or to account
for leap seconds. Such changes would cause errors in the measurement of
time intervals for those time services that use the absolute value of
the clock."
clock and timer functions
http://www.opengroup.org/onlinepubs/009695399/functions/clock_settime.html
"If the value of the CLOCK_REALTIME clock is set via clock_settime(),
the new value of the clock shall be used to determine the time of
expiration for absolute time services based upon the CLOCK_REALTIME
clock. This applies to the time at which armed absolute timers expire.
If the absolute time requested at the invocation of such a time service
is before the new value of the clock, the time service shall expire
immediately as if the clock had reached the requested time normally."
"The effect of setting a clock via clock_settime() on armed per-process
timers associated with a clock other than CLOCK_REALTIME is
implementation-defined."
Here's some background on what I'm trying to do:
My app expects a stream of UDP datagrams containing compressed video.
These datagrams have been time stamped by the source using a high
resolution clock.
The video stream is played out in real-time. I buffer several packets,
then repeatedly arm a one-shot timer (with TIMER_ABSTIME set) to wait
until it's time to send the oldest packet.
AFAIU, if I use the CLOCK_REALTIME clock in my app, and if the sysadmin
changes the date, hell will break loose in my app. Hence my need for
CLOCK_MONOTONIC.
On a related note, I see that do_gettimeofday() in kernel/timer.c
actually gets a struct timespec from __get_realtime_clock_ts() then
throws bits away in the ns to us conversion. The integer division is
also a waste of cycles in my case, as I want the result in ns.
I suppose that if I change sock_get_timestamp() in core/sock.c to call
__get_realtime_clock_ts() instead of do_gettimeofday() I'll break 50
billion applications (ping? traceroute?) that expect a struct timeval?
Anyway, thanks for reading this far.
Regards.
Mr Anonymous,
>
> My app expects a stream of UDP datagrams containing compressed video.
> These datagrams have been time stamped by the source using a high
> resolution clock.
Why do you need another time stamp them?
>
> The video stream is played out in real-time. I buffer several packets,
> then repeatedly arm a one-shot timer (with TIMER_ABSTIME set) to wait
> until it's time to send the oldest packet.
>
> AFAIU, if I use the CLOCK_REALTIME clock in my app, and if the sysadmin
> changes the date, hell will break loose in my app.
Only if your app cannot handle time going backwards.
> I suppose that if I change sock_get_timestamp() in core/sock.c to call
> __get_realtime_clock_ts() instead of do_gettimeofday() I'll break 50
> billion applications (ping? traceroute?) that expect a struct timeval?
Not that many, but some probably.
Letting the kernel do the time stamping is usually quite useless anyways.
You can as well do it in your application when you receive it. After all
you're interested in when you can read the packet in your app,
not when the driver processes it.
The kernel time stamping is mainly for sniffing applications that want
to know exactly what's going on on the wire. But even then it's fairly
arbitary because it can be a long time between the packet arriving
on the PHY and being processed by the driver.
-Andi
Andi Kleen wrote:
> John wrote:
>
>> My app expects a stream of UDP datagrams containing compressed video.
>> These datagrams have been time stamped by the source using a high
>> resolution clock.
>
> Why do you need another time stamp then?
Source and receiver do not share a common clock. Thus, I have to monitor
clock skew. If I simply use the time stamps produced by the source, my
buffer will eventually underflow or overflow.
As far as I have seen, clock skew is on the order of ?10^-5, i.e. when
the source thinks 100000 seconds have elapsed, the receiver thinks
100001 seconds have elapsed.
>> The video stream is played out in real-time. I buffer several packets,
>> then repeatedly arm a one-shot timer (with TIMER_ABSTIME set) to wait
>> until it's time to send the oldest packet.
>>
>> AFAIU, if I use the CLOCK_REALTIME clock in my app, and if the sysadmin
>> changes the date, hell will break loose in my app.
>
> Only if your app cannot handle time going backwards.
My app cannot handle time going backwards.
The algorithm looks like:
/* Start transmission, record start date */
NEXT_DEADLINE = now;
while ( 1 )
{
send(oldest_packet);
recv_available_packets();
/* compute next deadline */
NEXT_DEADLINE += time_to_wait_until_next_packet();
sleep_until(NEXT_DEADLINE);
}
If the clock changes under me, I'm toast.
>> I suppose that if I change sock_get_timestamp() in core/sock.c to call
>> __get_realtime_clock_ts() instead of do_gettimeofday() I'll break 50
>> billion applications (ping? traceroute?) that expect a struct timeval?
>
> Not that many, but some probably.
Another option would be to change sock_get_timestamp() to call
ktime_get_ts() instead of do_gettimeofday() and convert ns to us.
i.e. I still return a struct timeval, but using CLOCK_MONOTONIC.
> Letting the kernel do the time stamping is usually quite useless anyways.
> You can as well do it in your application when you receive it. After all
> you're interested in when you can read the packet in your app,
> not when the driver processes it.
I use clock_nanosleep() to sleep until it's time to send the next
packet. I check for packets when I wake up. I need the kernel to time
stamp the packets because I was sleeping when they reached the system.
I try to do it this way because I read this:
http://rt.wiki.kernel.org/index.php/High_resolution_timers
"Note that (clock_)nanosleep functions do not suffer from this problem
as the wakeup function at timer expiry is executed in the context of the
high resolution timer interrupt. If an application is not using
asynchronous signal handlers, it is recommended to use the
clock_nanosleep() function with the TIMER_ABSTIME flag set instead of
waiting for the periodic timer signal delivery. The application has to
maintain the absolute expiry time for the next interval itself, but this
is a lightweight operation of adding and normalizing two struct timespec
values. The benefit is significantly lower maximum latencies (~50us) and
less OS overhead in general."
Regards.
John wrote:
> Andi Kleen wrote:
>
>> John wrote:
>>
>>> My app expects a stream of UDP datagrams containing compressed video.
>>> These datagrams have been time stamped by the source using a high
>>> resolution clock.
>>
>> Why do you need another time stamp then?
>
> Source and receiver do not share a common clock. Thus, I have to monitor
> clock skew. If I simply use the time stamps produced by the source, my
> buffer will eventually underflow or overflow.
>
> As far as I have seen, clock skew is on the order of ?10^-5, i.e. when
> the source thinks 100000 seconds have elapsed, the receiver thinks
> 100001 seconds have elapsed.
I've considered running NTP to synchronize the clocks, but I'm afraid it
wouldn't help, for several reasons:
1) The 40.5 MHz clock used to timestamp datagrams at the source is
located on a PCI board. I have read access to a 32-bit counter that is
incremented every cycle (somewhat like the TSC on x86).
2) Source and receiver are inside a VPN, and do not have access to the
Internet. I don't think I can keep NTP happy if it can't talk to any NTP
servers.
(I'm not snipping the rest of the message in case someone is willing
to comment.)
>>> The video stream is played out in real-time. I buffer several packets,
>>> then repeatedly arm a one-shot timer (with TIMER_ABSTIME set) to wait
>>> until it's time to send the oldest packet.
>>>
>>> AFAIU, if I use the CLOCK_REALTIME clock in my app, and if the
>>> sysadmin changes the date, hell will break loose in my app.
>>
>> Only if your app cannot handle time going backwards.
>
> My app cannot handle time going backwards.
>
> The algorithm looks like:
>
> /* Start transmission, record start date */
> NEXT_DEADLINE = now;
>
> while ( 1 )
> {
> send(oldest_packet);
> recv_available_packets();
> /* compute next deadline */
> NEXT_DEADLINE += time_to_wait_until_next_packet();
> sleep_until(NEXT_DEADLINE);
> }
>
> If the clock changes under me, I'm toast.
>
>>> I suppose that if I change sock_get_timestamp() in core/sock.c to
>>> call __get_realtime_clock_ts() instead of do_gettimeofday() I'll
>>> break 50 billion applications (ping? traceroute?) that expect a
>>> struct timeval?
>>
>> Not that many, but some probably.
>
> Another option would be to change sock_get_timestamp() to call
> ktime_get_ts() instead of do_gettimeofday() and convert ns to us.
>
> i.e. I still return a struct timeval, but using CLOCK_MONOTONIC.
>
>> Letting the kernel do the time stamping is usually quite useless
>> anyways. You can as well do it in your application when you receive
>> it. After all you're interested in when you can read the packet in
>> your app, not when the driver processes it.
>
> I use clock_nanosleep() to sleep until it's time to send the next
> packet. I check for packets when I wake up. I need the kernel to time
> stamp the packets because I was sleeping when they reached the system.
>
> I try to do it this way because I read this:
>
> http://rt.wiki.kernel.org/index.php/High_resolution_timers
>
> "Note that (clock_)nanosleep functions do not suffer from this problem
> as the wakeup function at timer expiry is executed in the context of the
> high resolution timer interrupt. If an application is not using
> asynchronous signal handlers, it is recommended to use the
> clock_nanosleep() function with the TIMER_ABSTIME flag set instead of
> waiting for the periodic timer signal delivery. The application has to
> maintain the absolute expiry time for the next interval itself, but this
> is a lightweight operation of adding and normalizing two struct timespec
> values. The benefit is significantly lower maximum latencies (~50us) and
> less OS overhead in general."