Hi Dipen,
To support Intel timed i/o driver, it was suggested by Linux community to enhance hte framework.
However, we see some limitations to support complete Intel timed i/o device.
1. The current framework supports Nvidia IP, which has two IP blocks (hw timestamping engine interfaced with GPIO)
Intel timed i/o is a single IP block handling multiple functionalities like:
a. Input timestamping with event counter.
b. Timed output - single shot or periodic pulse train.
Uses TSC(Timestamp counter) for timestamp or generate events, which could be translated to system time.
c. Implement PPS functionality to export time.
This requires new functionality(interface) to be developed in hte framework specific to timed-io device.
2. The current hte framework has a provider and consumer concept.
Consumer is responsible for user space interaction.
Currently Nvidia is using GPIO for input timestamping (by adding hw timestamps in gpiolib-cdev.c)
For Intel timed i/o functionalities, current gpio user interfaces cannot support event counter
or output modes.
Rather than jigging hte consumer into other subsystems to support timed i/o device.
Any possibility of developing a native consumer in hte subsystem, which could handle user space interactions for timestamping engines.
I cannot think of an existing subsystem that handles Intel timed i/o functionalities 1a, 1b and 1c (mentioned above).
Regards,
Pandith
On 11/17/22 10:47 AM, N, Pandith wrote:
> Hi Dipen,
>
> To support Intel timed i/o driver, it was suggested by Linux community to enhance hte framework.
> However, we see some limitations to support complete Intel timed i/o device.
>
> 1. The current framework supports Nvidia IP, which has two IP blocks (hw timestamping engine interfaced with GPIO)
> Intel timed i/o is a single IP block handling multiple functionalities like:
>
> a. Input timestamping with event counter.
> b. Timed output - single shot or periodic pulse train.
> Uses TSC(Timestamp counter) for timestamp or generate events, which could be translated to system time.
> c. Implement PPS functionality to export time.
Isn't 1c similar to 1b, where IO (mostly GPIO) is programmed to toggle/periodic pulse train at 1s.
>
> This requires new functionality(interface) to be developed in hte framework specific to timed-io device.
I can see 1a is straight case for the HTE.
For 1b, the timestamp part can be added as hte provider. I see opportunity to enhance hte framework to provide
translation facility between the domain, system time in this case. However programming interface to facilitate
timed IO output can not fit into HTE the way it is right now. May be one possible way is to enchance HTE with API something like
hte_configure_timestamp_periodic/timed could be possible in which case HTE does something more than just timestamping the event.
I have to see how in GPIO case that proposed API works out, if it will bypass gpio framework etc...
Adding Linus W into the discussion....
>
> 2. The current hte framework has a provider and consumer concept.
> Consumer is responsible for user space interaction.
> Currently Nvidia is using GPIO for input timestamping (by adding hw timestamps in gpiolib-cdev.c)
>
> For Intel timed i/o functionalities, current gpio user interfaces cannot support event counter
> or output modes.
Can you elaborate on event counter and output mode?
> Rather than jigging hte consumer into other subsystems to support timed i/o device.
> Any possibility of developing a native consumer in hte subsystem, which could handle user space interactions for timestamping engines.
yes, feel free to send patches to me and cc [email protected], I guess you can register your IP as one of the provider.
To explain why GPIO was treated specially, there was already a
user facing framework i,e gpio-cdev and range of userspace tools which could be leveraged for the HTE GPIO consumers. However this does
not prevent kernel space GPIO HTE consumer from using HTE core directly.
>
> I cannot think of an existing subsystem that handles Intel timed i/o functionalities 1a, 1b and 1c (mentioned above).
>
> Regards,
> Pandith
>
>
>
>
Best Regards,
Dipen Patel
On Sat, Nov 19, 2022 at 3:33 AM Dipen Patel <[email protected]> wrote:
> On 11/17/22 10:47 AM, N, Pandith wrote:
> > To support Intel timed i/o driver, it was suggested by Linux community to enhance hte framework.
> > However, we see some limitations to support complete Intel timed i/o device.
(...)
> > b. Timed output - single shot or periodic pulse train.
> > Uses TSC(Timestamp counter) for timestamp or generate events, which could be translated to system time.
(...)
> For 1b, the timestamp part can be added as hte provider. I see opportunity to enhance hte framework to provide
> translation facility between the domain, system time in this case. However programming interface to facilitate
> timed IO output can not fit into HTE the way it is right now. May be one possible way is to enchance HTE with API something like
> hte_configure_timestamp_periodic/timed could be possible in which case HTE does something more than just timestamping the event.
>
> I have to see how in GPIO case that proposed API works out, if it will bypass gpio framework etc...
>
> Adding Linus W into the discussion....
I think I already answered with some open ended questions in discussion
with Christopher S. Hall:
https://lore.kernel.org/linux-iio/CACRpkdbeQ_67V3jkw_-KfTwe54TxrK_LA7N8Nwj1qEpTELN9dQ@mail.gmail.com/
https://lore.kernel.org/linux-iio/CACRpkda63TNWLdTjY+_xxXb4df4qCZi1EaXL3pXK=+Hon-0RLQ@mail.gmail.com/
https://lore.kernel.org/linux-iio/CACRpkdYxverx-KsG9URrh5qkEFXDknZKCE6RM555mjOuC--vMg@mail.gmail.com/
I think Intel is in a bit of uncomfortable design space with this thing that
isn't fitting properly anywhere. Dunno what to do about that :/
One thing I didn't get a clear answer for is what kind of use cases the
timed GPIO is for. I was suspecting things like stepper motors. I was
suspecting at one point some kind of sync pulse to something. But
no clear answer.
It is useful to know the usecase to be able to figure out the appropriate
subsystem. It might be that GPIO with some custom ioctl()s for
the gpiodev is the right answer to some of the question, as outlined
above.
and I see Dipen has some answers:
Yours,
Linus Walleij
> -----Original Message-----
> From: Dipen Patel <[email protected]>
> Sent: Saturday, November 19, 2022 8:04 AM
> To: N, Pandith <[email protected]>; [email protected]
> Cc: [email protected]; Hall, Christopher S
> <[email protected]>; Gross, Mark <[email protected]>;
> Sangannavar, Mallikarjunappa <[email protected]>;
> D, Lakshmi Sowjanya <[email protected]>; T R, Thejesh Reddy
> <[email protected]>; [email protected];
> [email protected]; [email protected]
> Subject: Re: Intel timed i/o driver in HTE
>
> On 11/17/22 10:47 AM, N, Pandith wrote:
> > Hi Dipen,
> >
> > To support Intel timed i/o driver, it was suggested by Linux community to
> enhance hte framework.
> > However, we see some limitations to support complete Intel timed i/o
> device.
> >
> > 1. The current framework supports Nvidia IP, which has two IP blocks
> > (hw timestamping engine interfaced with GPIO) Intel timed i/o is a single IP
> block handling multiple functionalities like:
> >
> > a. Input timestamping with event counter.
> > b. Timed output - single shot or periodic pulse train.
> > Uses TSC(Timestamp counter) for timestamp or generate events,
> which could be translated to system time.
> > c. Implement PPS functionality to export time.
> Isn't 1c similar to 1b, where IO (mostly GPIO) is programmed to
> toggle/periodic pulse train at 1s.
> >
Yes, Functionally they are quite similar. 1c is relatively slow where each event is scheduled by the driver to be aligned the system clock.
1b is relatively fast (in periodic mode). we are checking to support this is in a single interface.
> > This requires new functionality(interface) to be developed in hte
> framework specific to timed-io device.
> I can see 1a is straight case for the HTE.
Yes, current interfaces can support input mode. Since few Intel timed i/o devices do not have interrupt support.
There is a need to enhance current framework, to support polling mode.
>
> For 1b, the timestamp part can be added as hte provider. I see opportunity
> to enhance hte framework to provide translation facility between the
> domain, system time in this case. However programming interface to
> facilitate timed IO output can not fit into HTE the way it is right now. May be
> one possible way is to enchance HTE with API something like
> hte_configure_timestamp_periodic/timed could be possible in which case
> HTE does something more than just timestamping the event.
This is the possibility we wanted to check, to accommodate different output modes.
Something like hte_configure_timestamp or hte_request_pulse.
>
> I have to see how in GPIO case that proposed API works out, if it will bypass
> gpio framework etc...
>
> Adding Linus W into the discussion....
>
> >
> > 2. The current hte framework has a provider and consumer concept.
> > Consumer is responsible for user space interaction.
> > Currently Nvidia is using GPIO for input timestamping (by adding hw
> > timestamps in gpiolib-cdev.c)
> >
> > For Intel timed i/o functionalities, current gpio user interfaces
> > cannot support event counter or output modes.
> Can you elaborate on event counter and output mode?
In 1a, event counter holds the input trigger count on configured pin.
Output mode is case 1b and 1c.
>
> > Rather than jigging hte consumer into other subsystems to support timed
> i/o device.
> > Any possibility of developing a native consumer in hte subsystem, which
> could handle user space interactions for timestamping engines.
> yes, feel free to send patches to me and cc [email protected], I
> guess you can register your IP as one of the provider.
>
This native hte consumer is also intended to handle ioctl based user space interaction.
Use the services of enhanced hte provider (with a new char device type creation).
> To explain why GPIO was treated specially, there was already a user facing
> framework i,e gpio-cdev and range of userspace tools which could be
> leveraged for the HTE GPIO consumers. However this does not prevent
> kernel space GPIO HTE consumer from using HTE core directly.
>
Since the current gpio framework is inadequate to periodic output modes.
I thought to disentangle from gpio and develop a new hte consumer. Something like hte-libcdev.c
> >
> > I cannot think of an existing subsystem that handles Intel timed i/o
> functionalities 1a, 1b and 1c (mentioned above).
> >
> > Regards,
> > Pandith
> >
> >
> >
> >
> Best Regards,
> Dipen Patel
On Wed, Nov 23, 2022 at 3:38 PM N, Pandith <[email protected]> wrote:
> Since the current gpio framework is inadequate to periodic output modes.
> I thought to disentangle from gpio and develop a new hte consumer. Something like hte-libcdev.c
Maybe, if Dipen wants it.
Or maybe it needs its own subsystem.
Or maybe it should be with whatever is using these pulse trains.
Again: what is it used for? If we have no idea what it is supposed to
be used for, we cannot place it right, even less test it.
Yours,
Linus Walleij
> -----Original Message-----
> From: Linus Walleij <[email protected]>
> Sent: Thursday, November 24, 2022 2:53 AM
> To: N, Pandith <[email protected]>
> Cc: Dipen Patel <[email protected]>; [email protected]; Hall,
> Christopher S <[email protected]>; Gross, Mark
> <[email protected]>; Sangannavar, Mallikarjunappa
> <[email protected]>; D, Lakshmi Sowjanya
> <[email protected]>; T R, Thejesh Reddy
> <[email protected]>; [email protected];
> [email protected]
> Subject: Re: Intel timed i/o driver in HTE
>
> On Wed, Nov 23, 2022 at 3:38 PM N, Pandith <[email protected]> wrote:
>
> > Since the current gpio framework is inadequate to periodic output modes.
> > I thought to disentangle from gpio and develop a new hte consumer.
> > Something like hte-libcdev.c
>
> Maybe, if Dipen wants it.
> Or maybe it needs its own subsystem.
> Or maybe it should be with whatever is using these pulse trains.
>
> Again: what is it used for? If we have no idea what it is supposed to be used
> for, we cannot place it right, even less test it.
>
Intel timed i/o is mainly intended for couple of functionalities.
1. Input event capture with timestamping
2. Generate single shot or periodic pulse train
Use cases :
a. This has multiple uses like capturing trigger time and event count
b. Calculate periodicity in case of periodic signal on input pin
c. Most importantly, precise time synchronization between devices/sub-systems
Ex : Share precise time from a GPS receiver to the network or
Synchronize processor clock with external signal.
> Yours,
> Linus Walleij
Regards,
Pandith
On 11/23/22 1:22 PM, Linus Walleij wrote:
> On Wed, Nov 23, 2022 at 3:38 PM N, Pandith <[email protected]> wrote:
>
>> Since the current gpio framework is inadequate to periodic output modes.
>> I thought to disentangle from gpio and develop a new hte consumer. Something like hte-libcdev.c
>
> Maybe, if Dipen wants it.
> Or maybe it needs its own subsystem.
> Or maybe it should be with whatever is using these pulse trains.
>
Few things I want to know from Pandith as below:
1. Does timed io only meant for GPIO or other signals? if other signals, what type
of signals?
2. In the output mode case where for example, timed io IP outputs 1ms GPIO pulse,
does it also timestamp it for consumers to retrieve later?
3. Can you share rough idea you have in mind for the hte-libcdev?
> Again: what is it used for? If we have no idea what it is supposed to
> be used for, we cannot place it right, even less test it.
I think it can possibly act as time sync server as one of the use case.
>
> Yours,
> Linus Walleij
On Tue, Nov 29, 2022 at 7:10 PM N, Pandith <[email protected]> wrote:
> Intel timed i/o is mainly intended for couple of functionalities.
> 1. Input event capture with timestamping
I understand this part, and it is handled by the HTE subsystem.
> 2. Generate single shot or periodic pulse train
(...)
> c. Most importantly, precise time synchronization between devices/sub-systems
> Ex : Share precise time from a GPS receiver to the network or
> Synchronize processor clock with external signal.
So I think this is the actual use case of the output mode. The
pulse train output by HTE is to share precise time to GPS receivers.
So what about putting that part into drivers/gnss?
GNSS Global Navigation Satellite Subsystem like GPS etc
If this is the only usecase, that is where it should go, along
with the serial or whatever transport driver is used with the
GPS. I don't think this is a generic functionality (such as
GPIO) at all, but rather a very application-specific use case
which will only be used for GPS time synchronization.
If the timed output has other use cases - and I mean HAVE
other use cases, not COULD HAVE other use cases - then
we can discuss generalizations. There is no point of designing
upfront abstractions that never get used.
Yours,
Linus Walleij
> -----Original Message-----
> From: Dipen Patel <[email protected]>
> Sent: Wednesday, November 30, 2022 8:54 AM
> To: Linus Walleij <[email protected]>; N, Pandith
> <[email protected]>
> Cc: [email protected]; Hall, Christopher S
> <[email protected]>; Gross, Mark <[email protected]>;
> Sangannavar, Mallikarjunappa <[email protected]>;
> D, Lakshmi Sowjanya <[email protected]>; T R, Thejesh Reddy
> <[email protected]>; [email protected];
> [email protected]
> Subject: Re: Intel timed i/o driver in HTE
>
> On 11/23/22 1:22 PM, Linus Walleij wrote:
> > On Wed, Nov 23, 2022 at 3:38 PM N, Pandith <[email protected]>
> wrote:
> >
> >> Since the current gpio framework is inadequate to periodic output
> modes.
> >> I thought to disentangle from gpio and develop a new hte consumer.
> >> Something like hte-libcdev.c
> >
> > Maybe, if Dipen wants it.
> > Or maybe it needs its own subsystem.
> > Or maybe it should be with whatever is using these pulse trains.
> >
> Few things I want to know from Pandith as below:
> 1. Does timed io only meant for GPIO or other signals? if other signals, what
> type of signals?
This could be time sync signals or periodic/single pulse input for timestamp support
> 2. In the output mode case where for example, timed io IP outputs 1ms GPIO
> pulse, does it also timestamp it for consumers to retrieve later?
No. In output mode, either single shot or periodic pulse train is generated.
But timed i/o does keep timestamp data in input mode.
> 3. Can you share rough idea you have in mind for the hte-libcdev?
a. This layer can provide ioctl based user interactions for timestamp services (input and output mode)
b. char device creation for timestamping hardware. Something like /dev/hteX
c. Develop a native hte consumer, which handles user space interactions of hte subsystem
>
> > Again: what is it used for? If we have no idea what it is supposed to
> > be used for, we cannot place it right, even less test it.
> I think it can possibly act as time sync server as one of the use case.
> >
> > Yours,
> > Linus Walleij
On Fri, Dec 2, 2022 at 4:00 AM N, Pandith <[email protected]> wrote:
> > 1. Does timed io only meant for GPIO or other signals? if other signals, what
> > type of signals?
> This could be time sync signals or periodic/single pulse input for timestamp support
(...)
> b. char device creation for timestamping hardware. Something like /dev/hteX
No way this goes to userspace if the usecase is synchronizing GPS time.
It should be in the kernel, where the timebase of the system is.
We already created drivers/gnss because power, clock etc management
need to be close to the hardware, in the kernel. Don't try to push this
up to userspace, group it with the rest of the kernel GPS handling.
If your GPS vendor doesn't want the kernel to talk directly to the GPS,
bad luck, because the kernel definitely should in this case. The kernel
has the best time source and the shortest access path to it, so there
you go.
The GPS vendors have already created enough of a mess by not having
open documentation for their hardware, if this is an incentive for them to
be more open if they want proper time sync that is *good*.
Yours,
Linus Walleij
Hi Linus,
Thanks for your response.
On Saturday, December 03, 2022 1:50 AM Linus Walleij <[email protected]> wrote:
> To: N, Pandith <[email protected]>; Johan Hovold <[email protected]>
> Subject: Re: Intel timed i/o driver in HTE
>
> On Fri, Dec 2, 2022 at 4:00 AM N, Pandith <[email protected]> wrote:
>
> > > 1. Does timed io only meant for GPIO or other signals? if other signals, what
> > > type of signals?
> > This could be time sync signals or periodic/single pulse input for timestamp support
> (...)
> > b. char device creation for timestamping hardware. Something like /dev/hteX
>
> No way this goes to userspace if the usecase is synchronizing GPS time.
The Timed I/O device is for timestamped input and output only. There are
currently two dedicated single-ended signals on recent Intel platforms.
It is not a GPS device. It does not belong in the GNSS subsystem. We use
that as an example application that you had asked for. We propose any
device that provides a PPS signal out can be used to set the system
clock using Timed I/O.
For PPS input, we plan to use the PPS subsystem.
At a high-level:
The application configures the pin for PPS input
A device in created /dev/ppsX
while 1:
The timed I/O device captures / timestamps a pulse from an
external PPS provider
The timestamp is translated to system time(TN)
A PPS event is generated using pps_event(TN)
Another application like PHC2SYS or Chrony consumes the timestamps from
the PPS device and disciplines the system clock.
Currently, most - maybe all - PPS clients (drivers/pps/clients) capture
timestamps in software (pps_get_ts()). We can do at least 50x better
timestamping using hardware. The timestamp accuracy is in the range of
a few 10s of nanoseconds. I think this is a good thing.
> It should be in the kernel, where the timebase of the system is.
It is.
> We already created drivers/gnss because power, clock etc management
> need to be close to the hardware, in the kernel. Don't try to push this
> up to userspace, group it with the rest of the kernel GPS handling.
> The GPS vendors have already created enough of a mess by not having
> open documentation for their hardware, if this is an incentive for them to
> be more open if they want proper time sync that is *good*.
Again we provided an example application. Let's not get "hung up" on GPS.
There are many GPS receivers that produce a PPS output. There is some
discussion of the interface here if there is doubt such a thing exists:
https://electronics.stackexchange.com/questions/30750/why-do-gps-receivers-have-a-1-pps-output
If I may, I would like to re-focus the discussion. The question we want
to answer in this thread is whether it makes sense to modify the HTE
subsystem to accommodate our device and whether it "belongs" there.
For some background, I proposed creating a new Timed IO device type at
LPC:
https://lpc.events/event/16/contributions/1387/
The feedback was - to paraphrase: "We cannot support a new device type
for a single vendor. Can you use the HTE subsystem?"
This is a reasonable question. HTE devices implement a similar
function - hardware timestamping - to our Timed IO device. It is the
only similar device time that I am aware of. However, large changes to
the subsystem would be required to support output, for example.
To summarize this and previous threads the Timed IO use case is
importing and exporting system time with about nanosecond precision.
We are able to do this because the clock we use to generate or
timestamp events using TSC which drives the system clock on Intel.
To see how TSC -> System Time conversion works on input:
kernel/time/timekeeping.c:get_device_system_crosststamp()
More specific use cases are:
- GPS / PPS input to discipline the system clock
- 1 Hz to 1 kHz out signal synchronized with the system clock for
diagnostic output. This is for safety critical systems where
unsynchronized clocks are dangerous.
I have also used PPS output to measure synchronization between
devices connected to the platform:
https://drive.google.com/file/d/1OKAOjRzZHlaNoHE7USXpOyIBU0_1EytU/view
This measures PTM performance, but TGPIO can also be used to as a
substitute for PTM to synchronize the system clock with the NIC. The
second (PTM substitute) is similar to the PPS input use.
To support this, we need PPS In / Out functions and up to 1 kHz output.
These all must be aligned with the CLOCK_REALTIME.
To "future proof" our device we also want to use the periodic output
function of the Timed IO device. Using this, we can output up to about
4 MHz but rather than scheduling each output event as in the previous
use cases we start a signal that is aligned to the system clock, but
since the frequency is that of the TSC, not CLOCK_REALTIME, there is
drift and we use a PI controller to maintain zero offset with
CLOCK_REALTIME.
We also want to be able to timestamp / generate single events.
Existing HTE already do the first.
Are these use cases that you are willing to support in the HTE
subsystem? We seem to be unable to dig into the implementation
without circling back to the use cases which I believe are
clearly defined.
Thanks,
Christopher Hall
Hi Christopher!
thanks for the lengthy explanation, I think I get it now.
As usual I have now a slight impostor syndrome for now knowing
what PPS is and what it is for despite it has its own subsystem
and all.
On Thu, Dec 8, 2022 at 5:52 AM Hall, Christopher S
<[email protected]> wrote:
> For PPS input, we plan to use the PPS subsystem.
Why are you not planning to use the PPS "generators" for output?
> The application configures the pin for PPS input
> A device in created /dev/ppsX
> while 1:
> The timed I/O device captures / timestamps a pulse from an
> external PPS provider
> The timestamp is translated to system time(TN)
> A PPS event is generated using pps_event(TN
>
> Another application like PHC2SYS or Chrony consumes the timestamps from
> the PPS device and disciplines the system clock.
This is great and to the point.
> Currently, most - maybe all - PPS clients (drivers/pps/clients) capture
> timestamps in software (pps_get_ts()). We can do at least 50x better
> timestamping using hardware. The timestamp accuracy is in the range of
> a few 10s of nanoseconds. I think this is a good thing.
So the plan is to let a driver in drivers/pps obtain a timed IO input
from the HTE subsystem with high resution, which is great.
> Again we provided an example application. Let's not get "hung up" on GPS.
> There are many GPS receivers that produce a PPS output.
OK I'm sorry for being so stubborn with that example.
> If I may, I would like to re-focus the discussion. The question we want
> to answer in this thread is whether it makes sense to modify the HTE
> subsystem to accommodate our device and whether it "belongs" there.
Yeah I agree.
> To summarize this and previous threads the Timed IO use case is
> importing and exporting system time with about nanosecond precision.
Yeah I think that point has not been the main focus actually:
maybe I'm especially dumb but it looked to me as posed for
"random events" in and "random pulsetrains" out. But in this
context it makes much more sense.
> We also want to be able to timestamp / generate single events.
> Existing HTE already do the first.
>
> Are these use cases that you are willing to support in the HTE
> subsystem? We seem to be unable to dig into the implementation
> without circling back to the use cases which I believe are
> clearly defined.
Sorry for the misunderstandings. This is really Dipen's decision.
If you ask me, it seems like you should begin with making a
PPS input using the HTE as back-end, then think about the next
thing.
If this *makes* *sense*, and is not over-generalizing the
timed input. I.e. if you expect people to be using it for some
random other line sampling unrelated to PPS. There is no
point in putting it into drivers/hte for random abstraction.
Otherwise by all means put the whole thing into drivers/pps
why not? If that is what it is for?
This *could* be one of those cases where the subsystem is not
a clear cut, and that does not matter because we are not especially
OCD about pigeon-holeing hardware into one subsystem and
one subsystem only.
Perhaps the output mode should just go into drivers/pps/generators
without any HTE back-end while the input mode uses
HTE?
The fact that this is one single HW block doesn't really matter
as long as you can share the hw access using something like
MFD or regmap in worst case.
Yours,
Linus Walleij