I'd be happy to get some feedback on this, whether or not it is a good
idea or even the right way to approach this problem. This is part of
the puzzle for solving the problem of frequency scaling on (primarily)
embedded systems where there are many clocks, (changeable)
relationships between clocks, device dependencies on clock
availability, and device constraints on frequency that appear and
disappear as devices are enabled and disabled.
This is a first draft (almost thinking-out-loud version), so please
read it as such... there is much room for improvement and all
suggestions are welcome. Release early, release often, right? Well,
this is an "early" specification...
Regards,
Jonas
Introduction and Motivation
===========================
A system may be comprised of multiple clocks comprising CPU clock, bus
clocks, and external device clocks. The relationships amongst these
may be complex; this is particularly true for SOC's. There is
currently no abstraction layer in place that allows for a reasonably
system-agnostic management of this clock hierarchy and usage of this
clock information.
Today we have cpuidle, cpufreq, and struct clk which are all trying to
manage system clocks and their frequencies. Unfortunately, it seems
that there is a lot of overlap between these three areas. Cpuidle is
optimal on a system with explicit idle states; however, on other
systems, it may be best just to drop the operating frequency as low as
possible while idling. Cpufreq can dynamically change the operating
frequency, but collides with cpuidle when there are no explicit idle
states; furthermore, it needs to keep the configuration of the
hierarchy of struct clk objects sane. The struct clk objects provide
frequency information to drivers that depend on a given clock;
however, there is no notification system in place so that drivers know
(and can react appropriately) to clock frequency changes.
I suggest a new API that manages the system clock hierarchy, allows
frequency operating constraints to be set and respected, and allows
frequency changes to be appropriately notified across the system.
This API could be used by cpuidle when driving the system to a low
frequency state while idling; it could be used by cpufreq to select
appropriate operating frequencies and to set a new frequency in a
device-friendly manner; and it could be used by device drivers to
specify operating constraints for the clock that it uses while
minimizing the knowledge required about the rest of the clock
hierarchy.
The inspiration for this model comes largely from the clock hierarchy
of the Samsung S3C2410 processor; there is surely room for
improvement, especially in areas concerning specific requirements of
other systems. Feedback is welcome; function names are rather long
and suggestions for good, shortened names would be particularly
appreciated.
Clocks
======
The relationships between system clocks and the devices that rely on
them may be complex.
i) Clocks may be standalone or may depend on another clock (its parent)
ii) A clock may relate to another by way of a multiplier or divider
iii) The multiplier/divider may change
iv) A clock's parent may change
v) While clock frequencies (or the relationship between clocks) are
being adjusted, one or more clocks may become unstable/invalid and
should not be used
vi) Power usage is related to clock frequency
vii) A clock may be turned off completely if it is no device is using it
viii) A clock may be constrained by device requirements to work within
a given frequency range, or with a set of discrete frequencies
ix) A clock is constrained by the constraints of its child-clocks
x) A clock may switch between using discrete frequencies and using
a continuous range of frequencies as device constraints are updated.
xi) Device constraints may change dynamically; e.g. a device may
disengage the clock when it is not doing work or a device may change
its working frequency range based on some power saving considerations
These requirements give need for an API that allows for a clock's
capabilities to be defined and for the changing state of a clock
signal to be reflected across the system.
------ API Suggestions ------
Since clock, clk, and ck are already used in the kernel, we need a new
name that identifies this clock abstraction. I'm calling it 'klocka'
(Swedish word for clock) for lack of something better at the moment.
The clock has its own physical limitations when unconstrained by the
limits of any other device. We need to be able to specify these.
Set operating frequency range for unconstrained clock
klocka_set_freq_range(klocka, min, max)
Set operating frequency table for unconstrained clock
klocka_set_freq_table(klocka, table)
Register clocks:
klocka_register(klocka_info)
klocka_unregister(klocka)
System Functions
================
These functions act on 'klocka' objects, querying capabilities and
making operating frequency selections, as necesasry.
klocka_get_by_name(const char* name)
Make a clock available to clients
klocka_enable(klocka)
Turn off clock (at least, virtually); should make devices the rely on
clock quiesce.
klocka_disable(klocka)
Query the clock. It must be possible to:
- Get lowest possible operating frequency: returns the lowest
frequency that can be used given all device constraints, including
constraints of child clocks
- Get highest possible operating frequency: return the highest
frequency that can be used given all device constraints, including
constraints of child clocks
- Get closest frequency: return a valid operating frequency closest to
the requested frequency given all constraints.
Returns true if a clock can work with a set of discrete frequencies
only; false if the frequency range is continuous
klocka_has_discrete_frequencies(klocka)
Query available frequencies. For discrete clocks:
klocka_next_higher_frequency(klocka, freq);
klocka_next_lower_frequency(klocka, freq);
klocka_closest_frequency(klocka, freq);
For discrete and continuous clocks:
klocka_min_frequency()
klocka_max_frequency()
Selecting a new frequency:
klocka_set_frequency(klocka, freq)
--> This fails if freq is invalid within current constraints
Query current frequency:
klocka_get_frequency()
Drivers
=======
Drivers may have clock rate requirements; the driver may:
i) work only within a given frequency range,
ii) work only with certain discrete frequencies.
Furthermore, these driver requirements may be variable depending on
operating conditions:
i) frequency required only when device in operation
ii) device frequency requirements may change with operating conditions
The 'klocka' API should provide an interface for drivers to specify
their frequency requirements.
------ API Suggestions ------
Add a device that depends on clock:
klocka_device_add(klocka, device)
--> This function might just be implicit in some of the other below
Some drivers require the clock provide a signal within a certain
frequency range in order to function. Allow driver to tell the clock
that.
klocka_device_set_freq_range(klocka, device, min, max)
Some drivers can work only frequencies from a discrete set. Allow
this set to be provided to the 'klocka' as a table.
klocka_device_set_freq_table(klocka, device, frequencies):
When device is not busy, the clock frequency may be unimportant; in
this case, the driver can "disengage" the clock, allowing the clock
frequency to drift outside the values acceptable to the device for
normal operation or allowing the clock to be turned off altogether in
order to save power:
klocka_device_disengage(klocka, device)
When the device needs the clock to be turned on or to respect the
device frequency requirements, it can "engage" the clock. This
function should return a value telling the driver whether the clock is
already ready within the device's operating constraints so that it can
resume right away, or whether the clock needs to be adjusted so that
the device should wait for the appropriate frequency change
notification before resuming.
klocka_device_engage(klocka, device)
When the clock frequency changes, the driver may need notification in
order to change the device parameters; allow the driver to register
notifiers for frequency changes:
klocka_device_register_notifiers(klocka, device, notifiers)
The frequency change notifiers are optional and depend on the device
requirements. Some devices can adjust automatically, some need to
inhibit device operation during the change, some just need to update
some regisiters and can do so on the fly.
Notifiers
---------
pre_change: the frequency is about to change; quiesce the device if
necessary. This notifier is called while the clock is still running
at the old frequency.
change: update the device to work with the new frequency. This
notifier is called during the change; some clocks may need to be
stopped while changing frequency, in which case the clock might not be
running at all here.
post_change: the frequency has now changed; continue operation at new
frequency. This notifier is called when the clock is running at the
new frequency.
clock_invalidated: the clock is about to become unstable/be turned
off; quiesce if necessary
clock_validated: the clock has become stable/turned on again
Examples
========
Clock side
----------
(This is based loosely on S3C2410 processor, but is intended to be abstract)
Clock hierarchy:
MPLL
|--> fclk
|--> [Divider=2] --> hclk (hclk = fclk/2)
|--> [Divider=2] --> pclk (pclk = hclk/2)
fclk frequency may change which results in hclk and pclk changing as
well. Users of all three clocks need to be notified.
---------------
MPLL
|--> fclk
|--> [Divider=1] --> hclk (hclk = fclk)
|--> [Divider=2] --> pclk (pclk = hclk/2)
Here the divider for hclk has changed, resulting in a change to both
hclk and pclk frequencies. Users of these two clocks need to be
notified.
---------------
MPLL
|--> fclk
|--> [Divider=2] --> hclk (hclk = fclk/2)
|--> [Divider=2] --> pclk (pclk = fclk/2)
Here pclk has been reparented. Users of pclk need to be notified of
frequency change.
---------------
Driver side
-----------
The driver gets the clock it is using:
clk = klocka_get_by_name("hclk")
Now the driver can add device constraints to the clock:
klocka_device_set_freq_range(clk, dev, 100, 200)
...and the driver can register frequency-change notifiers:
klocka_device_register_notifiers(clk, dev, notifiers)
When frequency changes, we can be certain that it is within the
constraints given earlier (i.e. 100 < freq < 200). Notifiers will be
called before, during, and after frequency change as this driver
requires this; for other driver, notification might not be necessary
at all or only the change notification might be necessary.
pre_change notification:
driver disables device during frequency change
change notification:
driver updates device registers for new frequency
post_change notification:
driver resumes operation
We never have to worry about what is happening in the underlying clock
hierarchy because we are only dependent on the clock "hclk".
cpufreq notifiers (for some drivers) go away because we do not really
care about the "global frequency" scaling; we only care specifically
about the "real" frequency of the clock we are using; the cpufreq
infrastructure can adjust this clock as necessary and be sure that all
the users of the clock are correctly notified.
-----------------
Early Draft Definitions
=======================
struct klocka_device {
struct device* dev;
int engaged;
long min_freq;
long max_freq;
struct list_head discrete_frequencies;
void (*freq_prechange)(struct klocka *klk, struct device *dev);
void (*freq_change)(struct klocka *klk, struct device *dev, newfreq, oldfreq);
void (*freq_postchange)(struct klocka *klk, struct device *dev);
void (*klocka_invalidated)(struct klocka *klk, struct device *dev);
void (*klocka_validated)(struct klocka *klk, struct device *dev);
};
struct klocka {
const char* name;
/* These should not be manipulated directly */
struct klocka* parent;
struct list_head children;
struct list_head siblings;
struct list_head engaged_devices;
struct list_head disengaged_devices;
/* */
void* data;
/* Architecture/platform specific functions */
long (*get_freq)(struct klocka *klocka, int flags);
int (*set_freq)(struct klocka *klocka, int flags);
...
};
Jonas Bonn wrote:
> I'd be happy to get some feedback on this, whether or not it is a good
> idea or even the right way to approach this problem. This is part of
> the puzzle for solving the problem of frequency scaling on (primarily)
> embedded systems where there are many clocks, (changeable)
> relationships between clocks, device dependencies on clock
> availability, and device constraints on frequency that appear and
> disappear as devices are enabled and disabled.
>
> This is a first draft (almost thinking-out-loud version), so please
> read it as such... there is much room for improvement and all
> suggestions are welcome. Release early, release often, right? Well,
> this is an "early" specification...
Hmm, how does this relate to the OMAP clock domain work? I read about
it on LWN and was impressed...
omap2-clock merge: <http://thread.gmane.org/gmane.linux.ports.arm.omap/9464>
Len's OLS notes: <http://lwn.net/Articles/292447/>
Regards
Alan
On Thu, Oct 30, 2008 at 10:41 AM, Jonas Bonn <[email protected]> wrote:
> I'd be happy to get some feedback on this, whether or not it is a good
> idea or even the right way to approach this problem. This is part of
> the puzzle for solving the problem of frequency scaling on (primarily)
> embedded systems where there are many clocks, (changeable)
> relationships between clocks, device dependencies on clock
> availability, and device constraints on frequency that appear and
> disappear as devices are enabled and disabled.
How does this compare to the framework in linux/include/linux/clk.h?
--
Jon Smirl
[email protected]
>
> Hmm, how does this relate to the OMAP clock domain work? I read about
> it on LWN and was impressed...
>
> omap2-clock merge: <http://thread.gmane.org/gmane.linux.ports.arm.omap/9464>
>
> Len's OLS notes: <http://lwn.net/Articles/292447/>
>
Thanks for the heads up... I will take a closer look at this. At
first glance it appears to be on a similar path.
/Jonas
>
> How does this compare to the framework in linux/include/linux/clk.h?
>
clk.h is pretty much just an infrastructure for "storing" clock data
and accessors... it does not provide:
i) Constraints on selectable frequencies of active devices
ii) Notifiers on frequency change, including changes due to events
such as reparenting
These are the big ones that cause headaches. When I want to switch
frequency, I have to check elsewhere which devices are active and
which frequencies they allow. When a frequency changes, I have to
manually make sure that the active devices are notified properly of
the change. These are the main things that my document tries to
consolidate into one framework. Like I said, I'm not sure it's the
right approach, that's why I requested feedback...
/Jonas
On Thu, Oct 30, 2008 at 12:27 PM, Jonas Bonn <[email protected]> wrote:
>>
>> How does this compare to the framework in linux/include/linux/clk.h?
>>
>
> clk.h is pretty much just an infrastructure for "storing" clock data
> and accessors... it does not provide:
>
> i) Constraints on selectable frequencies of active devices
> ii) Notifiers on frequency change, including changes due to events
> such as reparenting
>
> These are the big ones that cause headaches. When I want to switch
> frequency, I have to check elsewhere which devices are active and
> which frequencies they allow. When a frequency changes, I have to
> manually make sure that the active devices are notified properly of
> the change. These are the main things that my document tries to
> consolidate into one framework. Like I said, I'm not sure it's the
> right approach, that's why I requested feedback...
You could probably work those features into the existing clk framework.
clk_set_rate() could compute the constrains and return an error.
The API could be expanded with notifier support.
--
Jon Smirl
[email protected]
On Thu, Oct 30, 2008 at 12:39 PM, Jon Smirl <[email protected]> wrote:
> On Thu, Oct 30, 2008 at 12:27 PM, Jonas Bonn <[email protected]> wrote:
>>>
>>> How does this compare to the framework in linux/include/linux/clk.h?
>>>
>>
>> clk.h is pretty much just an infrastructure for "storing" clock data
>> and accessors... it does not provide:
>>
>> i) Constraints on selectable frequencies of active devices
>> ii) Notifiers on frequency change, including changes due to events
>> such as reparenting
>>
>> These are the big ones that cause headaches. When I want to switch
>> frequency, I have to check elsewhere which devices are active and
>> which frequencies they allow. When a frequency changes, I have to
>> manually make sure that the active devices are notified properly of
>> the change. These are the main things that my document tries to
>> consolidate into one framework. Like I said, I'm not sure it's the
>> right approach, that's why I requested feedback...
>
> You could probably work those features into the existing clk framework.
> clk_set_rate() could compute the constrains and return an error.
> The API could be expanded with notifier support.
BTW, most user of the clk framework are on ARM.
>
> --
> Jon Smirl
> [email protected]
>
--
Jon Smirl
[email protected]
>
> You could probably work those features into the existing clk framework.
> clk_set_rate() could compute the constrains and return an error.
> The API could be expanded with notifier support.
>
I started in this end, too, and got frequency change notification
working, at least. There are issues with mutexes that "might_sleep"
when calling set_rate from cpuidle driver, but nothing that can't be
fixed...
What drove me to document a new interface is the fact that there are
so many users of "struct clk" already, that it becomes conceptually
easier to dream up something new that stays out of the way, even if
that new thing becomes just a wrapper around the existing interface
(which I am well aware that it largely is) with some new
functionality. If we can bolt the new stuff onto the existing stuff,
then that's even better; however, I know that yesterday I was longing
to be able to start from scratch and "do it right"! After writing my
document, though, I think I have largely validated the model that's in
place...
That said, I still think there is value in the additional features.
/Jonas
On Thu, Oct 30, 2008 at 06:01:42PM +0100, Jonas Bonn wrote:
> > You could probably work those features into the existing clk framework.
> > clk_set_rate() could compute the constrains and return an error.
> > The API could be expanded with notifier support.
...
> What drove me to document a new interface is the fact that there are
> so many users of "struct clk" already, that it becomes conceptually
> easier to dream up something new that stays out of the way, even if
> that new thing becomes just a wrapper around the existing interface
BTW, on this point I don't know if you've seen Dmitry Baryshkov's work
on a generic implementation of the existing clk API.
>
> BTW, on this point I don't know if you've seen Dmitry Baryshkov's work
> on a generic implementation of the existing clk API.
>
No, I haven't... where could I find more information on this?
/Jonas
On Thu, Oct 30, 2008 at 03:41:17PM +0100, Jonas Bonn wrote:
> I'd be happy to get some feedback on this, whether or not it is a good
> idea or even the right way to approach this problem. This is part of
Something like this would certainly be very useful for audio clocking if
it could be made to fit - that can get complex, is partly off-SoC and
frequently has multiple interrelated clock sources available, often with
various hard to describe interdependencies with constraints coming from
multiple sources and changing dynamically at run time.
In general your problem statement looks fairly sane. I've not reviewed
this in detail but do think you should have a look at the OMAP stuff.
One thing to watch out for is that you'll find clocks doing things like
originating in one chip, going through another and being fed back into
the original chip.
On Thu, Oct 30, 2008 at 09:15:54PM +0100, Jonas Bonn wrote:
> > BTW, on this point I don't know if you've seen Dmitry Baryshkov's work
> > on a generic implementation of the existing clk API.
> No, I haven't... where could I find more information on this?
Here's at least one posting of the series:
http://lkml.org/lkml/2008/6/26/279
No idea if it was the most recent. I've also CCed in Dmitry here.
Hello all,
Thanks for all your feedback. It's been very helpful.
I have thought about this some more and it seems quite feasible to add
a reasonable device interface to the existing clk infrastructure to
allow for specifying device frequency constraints and for
notifications to devices when clock frequencies change. Below is a
description of some of the new API... it only accounts for continuous
frequency range constraints right now, but it is planned to allow for
frequency tables, as well.
I have implemented this interface for the S3C2410 and it provides the
functionality that I want, namely that clocks become first-class
citizens and can be adjusted without specific knowledge about all the
underlying devices that may or may not be using them at any given
time. Most of the code will fit nicely into the GENERIC_CLK
infrastructure as it's essentially just list manipulation and callback
invocation... but more on that another time as I'm mainly just
interested in feedback on the API at this point in time.
API follows...
Regards,
Jonas
/* Device functions */
enum {
CLK_FREQ_PRECHANGE,
CLK_FREQ_POSTCHANGE,
CLK_FREQ_FAILED,
CLK_FREQ_INVALID,
CLK_FREQ_VALID
};
/**
* clk_dev_set_freq_range - set bounds of continuous freq range constraints
* @clk: clock source
* @dev: device
* @min: minimum acceptable frequency in Hz
* @max: maximum acceptable frequency in Hz
*
* This sets minimum and maximum frequencies that are acceptable as input to
* the device. The device is guaranteed that the clock will respect these
* constraints as long as the device is "engaged" (see below) to the clock.
*
* After this function returns, the clock "knows" about the device, but the
* device is not automatically engaged to the clock (see below).
*
* Return success (0)
* -EINVAL if clock is physically incapable of providing any frequency within
* these bounds
* ******FIXME******: select good errno's to return for these cases
* -EWAIT_FOR_NOTIFCATION : the clock frequency must be adjusted to meet the
* new device constraints; the device can consider itself engaged
* to the clock, but should wait for a frequency change
* notification before enabling itself
* -EANOTHER_DEVICE_PREVENTS_THESE CONSTRAINTS:
* if the device is already engaged to the clock and tries to
* set a frequency range that does not intersect with the
* constraints of another _engaged_ device, then this errno
* will be returned; the device can disengage from the clock
* and then set the same constraints again, or it can remain
* engaged and try another set of constraints that hopefully
* will intersect with the constraints of other devices.
*/
int clk_dev_set_freq_range(struct clk* clk, struct device* dev,
unsigned long min, unsigned long max);
/**
* clk_dev_set_notifier - set callback for clock frequency changes
* @clk: clock source
* @dev: device
* @notifier: frequency change callback
*
* This sets a callback to be invoked when the clock's frequency is changing.
*
* After this function returns, the clock "knows" about the device, but the
* device is not automatically engaged to the clock (see below).
*
* The notifier gets called twice, once before and once after the clock has
* switched frequency. The device can take necessary measures depending on
* its own requirements to adjust to the new input frequency from the clock.
*
* The type indicates whether the change is about to happen or has just
* happened; or whether the attempt to physically change the clock rate failed.
* Possible values for type are CPU_FREQ_PRECHANGE, CPU_FREQ_POSTCHANGE, or
* CPU_FREQ_FAILED.
*
* If the type is CPU_FREQ_PRECHANGE, the clock is running at 'oldfreq'; for
* CPU_FREQ_POSTCHANGE, the clock is running at 'newfreq'.
*
* If the type is CPU_FREQ_FAILED, then the driver is expected to reconfigure
* itself to work with 'oldfreq'; this is important in the case where
the driver
* made changes already in the PRECHANGE stage. When CPU_FREQ_FAILED
* notification is called, the clock should be assumed to be running at
* oldfreq; newfreq is largely informational, but indicates to the driver which
* frequency the clock _failed_ to be set to and which frequency the driver
* may have made adjusts to in the PRECHANGE stage.
*
* Note that the notifier gets called for both engaged and disengaged devices.
*
* Returns success (0)
*/
int clk_dev_set_notifier(struct clk* clk,
struct device* dev,
void (*notifier)(struct clk *clk,
struct device *dev,
unsigned long newfreq,
unsigned long oldfreq,
int type));
/**
* clk_dev_engage - "connect" the device to the clock.
* @clk: clock source
* @dev: device
*
* This activates the frequency constraints of this device for the clock.
* After engaging the clock, the device is assumed to be actively using the
* clock; the device can assume that its frequency constraints will be
* respected by the clock.
*
* Returns:
* 0 : if clock already meets device constraints
* ******FIXME******: select good errno's to return for these cases
* -EWAIT_FOR_NOTIFCATION : the clock frequency must be adjusted to meet the
* device constraints; the device can consider itself engaged to
* the clock, but should wait for a frequency change notification
* before enabling itself
* -EANOTHER_DEVICE_PREVENTS_ENGAGEMENT: another device is already
engaged to the
* clock with constraints that do not intersect with this
* device's constraints; it will be impossible to engage at
* this time and the device remains unengaged (the device can
* change its constraints and try again, if the device spec
* allows)
*/
int clk_dev_engage(struct clk* clk, struct device* dev);
/**
* clk_dev_disengage - "disconnect" the device from the clock.
* @clk: clock source
* @dev: device
*
* This deactivates the frequency constraints of this device for the clock.
* After disengaging the device, the clock may run at frequencies outside of
* the device constraints or even be turned off completely. The device is
* assumed to be "not listening" to the clock anymore.
*
* Returns nothing; disengaging always succeeds
*/
void clk_dev_disengage(struct clk* clk, struct device* dev);
/**
* clk_max_freq - return maximum clock frequency under current constraints
* @clk:
*
* This function takes into account the frequency constraints of all _engaged_
* devices and all child clocks to return the highest frequency that the clock
* may run at.
*/
unsigned long clk_max_freq(struct clk* clk);
/**
* clk_min_freq - return minimum clock frequency under current constraints
* @clk:
*
* This function takes into account the frequency constraints of all _engaged_
* devices and all child clocks to return the lowest frequency that the clock
* may run at.
*/
unsigned long clk_min_freq(struct clk* clk);