A number of us had a face-to-face meeting in Dublin last week to talk
about DT maintainership and the fact that it simply isn't working right
now. Neither Rob nor I can keep up with the load and there are a lot of
poorly designed bindings appearing in the tree.
Device tree binding maintainership needs to be split off to a separate
group, and we've started with a few people willing to help, Pawel Moll,
Mark Rutland, Stephen Warren and Ian Campbell.
(BTW, even though I've already sent a patch adding that group
MAINTAINERS, this is not set in stone. Anyone else wanting to help
maintain should volunteer)
Another thing discussed is that we need to start validating DT schema
with an extension to dtc. Tomasz Figa has volunteered to do this work
and has support from his employer to spend time on it. What I'm hoping
to have is that the DT schema will get checked as part of the dts build
process so that any DT file that doesn't match the documented schema
will get flagged, and that the schema files will be human readable and
will double as documentation.
There is not yet any process for binding maintainership. We talked about
a few ideas, but they really need to be hashed out here on the mailing
list. A couple of the questions:
- How are bindings allowed to be merged? Through subsystem trees, or
only through the bindings tree?
- Through the bindings tree is more work but it will provide more
control.
- Through subsystem trees means drivers and bindings get merged
together.
- If we have a schema tool that reports errors on missing or
unapproved schema, then spliting the driver from the binding won't
matter that much.
- Do we need to differentiate between 'staging' and 'stable' bindings?
- What is the schedule for splitting the bindings and .dts files out of
the kernel?
- Ian Campbell is maintaining a DT bindings and .dts mirror tree which
should eventually become the 'master' for merging DT bindings.
g.
On Saturday 20 of July 2013 04:46:47 Grant Likely wrote:
> A number of us had a face-to-face meeting in Dublin last week to talk
> about DT maintainership and the fact that it simply isn't working right
> now. Neither Rob nor I can keep up with the load and there are a lot of
> poorly designed bindings appearing in the tree.
>
> Device tree binding maintainership needs to be split off to a separate
> group, and we've started with a few people willing to help, Pawel Moll,
> Mark Rutland, Stephen Warren and Ian Campbell.
>
> (BTW, even though I've already sent a patch adding that group
> MAINTAINERS, this is not set in stone. Anyone else wanting to help
> maintain should volunteer)
>
> Another thing discussed is that we need to start validating DT schema
> with an extension to dtc. Tomasz Figa has volunteered to do this work
> and has support from his employer to spend time on it. What I'm hoping
> to have is that the DT schema will get checked as part of the dts build
> process so that any DT file that doesn't match the documented schema
> will get flagged, and that the schema files will be human readable and
> will double as documentation.
>
> There is not yet any process for binding maintainership. We talked about
> a few ideas, but they really need to be hashed out here on the mailing
> list. A couple of the questions:
>
> - How are bindings allowed to be merged? Through subsystem trees, or
> only through the bindings tree?
> - Through the bindings tree is more work but it will provide more
> control.
> - Through subsystem trees means drivers and bindings get merged
> together.
> - If we have a schema tool that reports errors on missing or
> unapproved schema, then spliting the driver from the binding won't
> matter that much.
> - Do we need to differentiate between 'staging' and 'stable' bindings?
> - What is the schedule for splitting the bindings and .dts files out of
> the kernel?
> - Ian Campbell is maintaining a DT bindings and .dts mirror tree which
> should eventually become the 'master' for merging DT bindings.
I remember getting to a conclusion that:
- bindings should enter staging state after being introduced,
- from time to time a binding review meeting should take place (on IRC
possibly) and discuss which of introduced staging bindings are ready to
enter stable state,
- in stable state such binding would be considered an ABI.
>From remaining questions I remember:
- How should we mark bindings as staging and stable? (i.e.
documentation/schema files in different folders or something else?)
- Should we also split parts of dts/dtsi using staging bindings from
those using stable ones? (This could mean including stable dts/dtsi file
from inside unstable one, which would allow having a stable dts with basic
functionality that could be extended over the time after validating
staging bindings used in unstable part.)
As for my part, I'm now looking into existing infrastructure inside of dtc
to get some hints that would allow me to design the initial schema syntax
in a most dtc-friendly way. Give me a bit more time and I will then write
down everything I have and post to the ML to start a discussion.
Best regards,
Tomasz
On Sat, Jul 20, 2013 at 04:46:47AM +0100, Grant Likely wrote:
> A number of us had a face-to-face meeting in Dublin last week to talk
> about DT maintainership and the fact that it simply isn't working right
> now. Neither Rob nor I can keep up with the load and there are a lot of
> poorly designed bindings appearing in the tree.
>
> Device tree binding maintainership needs to be split off to a separate
> group, and we've started with a few people willing to help, Pawel Moll,
> Mark Rutland, Stephen Warren and Ian Campbell.
>
> (BTW, even though I've already sent a patch adding that group
> MAINTAINERS, this is not set in stone. Anyone else wanting to help
> maintain should volunteer)
Sounds good.
> Another thing discussed is that we need to start validating DT schema
> with an extension to dtc. Tomasz Figa has volunteered to do this work
> and has support from his employer to spend time on it. What I'm hoping
> to have is that the DT schema will get checked as part of the dts build
> process so that any DT file that doesn't match the documented schema
> will get flagged, and that the schema files will be human readable and
> will double as documentation.
Tomasz, please keep me in the loop on this. This sounds like exactly
what the "checks" infrastructure in dtc was designed for, but I never
had time to implement very much there. I'd definitely like to follow
progress here, though.
> There is not yet any process for binding maintainership. We talked about
> a few ideas, but they really need to be hashed out here on the mailing
> list. A couple of the questions:
>
> - How are bindings allowed to be merged? Through subsystem trees, or
> only through the bindings tree?
> - Through the bindings tree is more work but it will provide more
> control.
> - Through subsystem trees means drivers and bindings get merged
> together.
> - If we have a schema tool that reports errors on missing or
> unapproved schema, then spliting the driver from the binding won't
> matter that much.
> - Do we need to differentiate between 'staging' and 'stable' bindings?
> - What is the schedule for splitting the bindings and .dts files out of
> the kernel?
> - Ian Campbell is maintaining a DT bindings and .dts mirror tree which
> should eventually become the 'master' for merging DT bindings.
It seems to me that the kernel tree has become the informal repository
for board dts files is in itself a problem. It encourages people to
think the two are closely linked and that all that matters is that a
specific dts works with its corresponding kernel and vice versa,
rather than fdts being the general description they're supposed to be.
Not that I have much in the way of better ideas.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
Grant Likely wrote:
> Another thing discussed is that we need to start validating DT schema with an extension to dtc.
Is there a schema out there in the wild that exemplifies what you mean?
> Tomasz Figa has volunteered to do this work and has support from his employer
That's great news.
> to have is that the DT schema will get checked as part of the dts build
> process so that any DT file that doesn't match the documented schema
> will get flagged, and that the schema files will be human readable and
> will double as documentation.
No doubt DTS files are already the best documentation available for many targets. The vendor's technical reference manual describes how the hardware is supposed to work, but the DTS describes what actually does. Any errata that the vendor issues subsequent to publication of the TRM must be reflected in DTS, after all.
The schema-check idea reminds me of the W3C HTML validators:
http://validator.w3.org/
Since device-tree source looks a bit like XML (or maybe more like JSON), will be the schemas be similar in spirit to DTDs, and is it helpful to think of the validator in this spirit? Or will the checker be more like "gcc -Wall", since it will be invoked by a compiler?
-- Alison
Hi Alison,
On Monday 22 of July 2013 19:59:25 Chaiken, Alison wrote:
> Grant Likely wrote:
> > Another thing discussed is that we need to start validating DT schema
> > with an extension to dtc.
> Is there a schema out there in the wild that exemplifies what you mean?
Not really. The format of schemas is currently in design stage. I'm
currently rethinking some details of what I have in my mind. Give me some
more time and I will post an RFC to the ML with all that written down.
> > Tomasz Figa has volunteered to do this work and has support from his
> > employer
> That's great news.
>
> > to have is that the DT schema will get checked as part of the dts
> > build
> > process so that any DT file that doesn't match the documented schema
> > will get flagged, and that the schema files will be human readable and
> > will double as documentation.
>
> No doubt DTS files are already the best documentation available for many
> targets. The vendor's technical reference manual describes how the
> hardware is supposed to work, but the DTS describes what actually does.
> Any errata that the vendor issues subsequent to publication of the
> TRM must be reflected in DTS, after all.
I'm not sure about this. Device tree should describe what hardware it is,
not how it works, unless it is really necessary.
> The schema-check idea reminds me of the W3C HTML validators:
>
> http://validator.w3.org/
>
> Since device-tree source looks a bit like XML (or maybe more like JSON),
> will be the schemas be similar in spirit to DTDs, and is it helpful to
> think of the validator in this spirit? Or will the checker be more
> like "gcc -Wall", since it will be invoked by a compiler?
My idea is to implement compile time verification in dtc, so I guess it
will be more like the latter. Since dts is what dtc can already parse, my
plan is to keep the schemas in spirit to dts, just modifying/extending it
to allow specifying bindings with them, rather than static values.
Best regards,
Tomasz
> > Is there a schema out there in the wild that exemplifies what you mean?
>
> Not really. The format of schemas is currently in design stage. I'm
> currently rethinking some details of what I have in my mind. Give me some
> more time and I will post an RFC to the ML with all that written down.
...and...
> > The schema-check idea reminds me of the W3C HTML validators:
> >
> > http://validator.w3.org/
> >
> > Since device-tree source looks a bit like XML (or maybe more like JSON),
> > will be the schemas be similar in spirit to DTDs, and is it helpful to
> > think of the validator in this spirit? Or will the checker be more
> > like "gcc -Wall", since it will be invoked by a compiler?
>
> My idea is to implement compile time verification in dtc, so I guess it
> will be more like the latter. Since dts is what dtc can already parse, my
> plan is to keep the schemas in spirit to dts, just modifying/extending it
> to allow specifying bindings with them, rather than static values.
>
> Best regards,
> Tomasz
It is possible to add some-damn XML DTD parsing and
rule glomming even in DTC if that is what is wanted.
I think it will be more an issue of expressability.
That is, how can the desired style of rules be most
eloquently expressed, internalized and applied?
That will be the current challenge first.
jdl
On Monday 22 of July 2013 16:34:49 Jon Loeliger wrote:
> > > Is there a schema out there in the wild that exemplifies what you
> > > mean?
> >
> > Not really. The format of schemas is currently in design stage. I'm
> > currently rethinking some details of what I have in my mind. Give me
> > some more time and I will post an RFC to the ML with all that written
> > down.
> ...and...
I meant a bit longer "more time". It's late here and it's almost time to
go to bed.
I need to look a bit inside dtc to see what is already there, but expect
results rather later this week.
> > > The schema-check idea reminds me of the W3C HTML validators:
> > > http://validator.w3.org/
> > >
> > > Since device-tree source looks a bit like XML (or maybe more like
> > > JSON), will be the schemas be similar in spirit to DTDs, and is it
> > > helpful to think of the validator in this spirit? Or will the
> > > checker be more like "gcc -Wall", since it will be invoked by a
> > > compiler?
> >
> > My idea is to implement compile time verification in dtc, so I guess
> > it
> > will be more like the latter. Since dts is what dtc can already parse,
> > my plan is to keep the schemas in spirit to dts, just
> > modifying/extending it to allow specifying bindings with them, rather
> > than static values.
> >
> > Best regards,
> > Tomasz
>
> It is possible to add some-damn XML DTD parsing and
> rule glomming even in DTC if that is what is wanted.
IMHO it's more natural to describe device tree bindings using dts-like
format, if not just because of tools already existing, then at least from
readability point of view.
> I think it will be more an issue of expressability.
> That is, how can the desired style of rules be most
> eloquently expressed, internalized and applied?
>
> That will be the current challenge first.
Yes, it will and it is already.
There is no problem with simple rules, such as required or optional
properties with predefined names, this is pretty obvious.
Things start to become fun when you get to bindings like regulators or
clocks, when part of the binding is defined on generic level (-supply,
clocks, clock-names properties) and remaining part is specific to device
(XXX in XXX-supply, count and order of clocks and clock-names, strings
allowed in clock-names property). This kind of inheritance is likely to be
the biggest troublemaker.
Best regards,
Tomasz
On 07/22/2013 12:59 PM, Chaiken, Alison wrote:
> Grant Likely wrote:
>> Another thing discussed is that we need to start validating DT schema with an extension to dtc.
>
> Is there a schema out there in the wild that exemplifies what you mean?
>
>> Tomasz Figa has volunteered to do this work and has support from his employer
>
> That's great news.
>
>> to have is that the DT schema will get checked as part of the dts build
>> process so that any DT file that doesn't match the documented schema
>> will get flagged, and that the schema files will be human readable and
>> will double as documentation.
>
> No doubt DTS files are already the best documentation available for many targets.
A nit here:
The DTS files document what HW modules are present on a board, and
various associated parameters.
The bindings describe the format of those DTS files.
The validation here is ensuring that the DTS files are actually written
in a format that matches what the binding definition requires.
So, the issue isn't really documentation of the HW, but rather
documentation of the required DT format, and validating that the DTS
matches that.
On 07/22/2013 02:57 PM, Tomasz Figa wrote:
> On Monday 22 of July 2013 16:34:49 Jon Loeliger wrote:
>>> My idea is to implement compile time verification in dtc, so I guess it
>>> will be more like the latter. Since dts is what dtc can already parse,
>>> my plan is to keep the schemas in spirit to dts, just
>>> modifying/extending it to allow specifying bindings with them, rather
>>> than static values.
> Things start to become fun when you get to bindings like regulators or
> clocks, when part of the binding is defined on generic level (-supply,
> clocks, clock-names properties) and remaining part is specific to device
> (XXX in XXX-supply, count and order of clocks and clock-names, strings
> allowed in clock-names property). This kind of inheritance is likely to be
> the biggest troublemaker.
It's good you mentioned inheritance here. I believe that's one of the
key things. For example, the Tegra GPIO controller's binding is-a
GPIO-controller, and is-an interrupt-controller, and I imagine any
successful DT schema definition would very explicitly include that
information. Likewise, other nodes may be is-a GPIO-client (many times,
each parameterized with the property name that defines which GPIO you're
talking abot), and also may be is-an interrupt-client (with a similar
comment), etc.
On Mon, 22 Jul 2013 16:34:49 -0500, Jon Loeliger <[email protected]> wrote:
>
> > > Is there a schema out there in the wild that exemplifies what you mean?
> >
> > Not really. The format of schemas is currently in design stage. I'm
> > currently rethinking some details of what I have in my mind. Give me some
> > more time and I will post an RFC to the ML with all that written down.
>
> ...and...
>
> > > The schema-check idea reminds me of the W3C HTML validators:
> > >
> > > http://validator.w3.org/
> > >
> > > Since device-tree source looks a bit like XML (or maybe more like JSON),
> > > will be the schemas be similar in spirit to DTDs, and is it helpful to
> > > think of the validator in this spirit? Or will the checker be more
> > > like "gcc -Wall", since it will be invoked by a compiler?
> >
> > My idea is to implement compile time verification in dtc, so I guess it
> > will be more like the latter. Since dts is what dtc can already parse, my
> > plan is to keep the schemas in spirit to dts, just modifying/extending it
> > to allow specifying bindings with them, rather than static values.
> >
> > Best regards,
> > Tomasz
>
> It is possible to add some-damn XML DTD parsing and
/me runs screaming.
g.
On Tue, 23 Jul 2013 08:55:10 -0700, Stephen Warren <[email protected]> wrote:
> On 07/22/2013 02:57 PM, Tomasz Figa wrote:
> > On Monday 22 of July 2013 16:34:49 Jon Loeliger wrote:
>
> >>> My idea is to implement compile time verification in dtc, so I guess it
> >>> will be more like the latter. Since dts is what dtc can already parse,
> >>> my plan is to keep the schemas in spirit to dts, just
> >>> modifying/extending it to allow specifying bindings with them, rather
> >>> than static values.
>
> > Things start to become fun when you get to bindings like regulators or
> > clocks, when part of the binding is defined on generic level (-supply,
> > clocks, clock-names properties) and remaining part is specific to device
> > (XXX in XXX-supply, count and order of clocks and clock-names, strings
> > allowed in clock-names property). This kind of inheritance is likely to be
> > the biggest troublemaker.
>
> It's good you mentioned inheritance here. I believe that's one of the
> key things. For example, the Tegra GPIO controller's binding is-a
> GPIO-controller, and is-an interrupt-controller, and I imagine any
> successful DT schema definition would very explicitly include that
> information. Likewise, other nodes may be is-a GPIO-client (many times,
> each parameterized with the property name that defines which GPIO you're
> talking abot), and also may be is-an interrupt-client (with a similar
> comment), etc.
+1
On Sat, 20 Jul 2013 15:49:21 +0200, Tomasz Figa <[email protected]> wrote:
> On Saturday 20 of July 2013 04:46:47 Grant Likely wrote:
> > A number of us had a face-to-face meeting in Dublin last week to talk
> > about DT maintainership and the fact that it simply isn't working right
> > now. Neither Rob nor I can keep up with the load and there are a lot of
> > poorly designed bindings appearing in the tree.
> >
> > Device tree binding maintainership needs to be split off to a separate
> > group, and we've started with a few people willing to help, Pawel Moll,
> > Mark Rutland, Stephen Warren and Ian Campbell.
> >
> > (BTW, even though I've already sent a patch adding that group
> > MAINTAINERS, this is not set in stone. Anyone else wanting to help
> > maintain should volunteer)
> >
> > Another thing discussed is that we need to start validating DT schema
> > with an extension to dtc. Tomasz Figa has volunteered to do this work
> > and has support from his employer to spend time on it. What I'm hoping
> > to have is that the DT schema will get checked as part of the dts build
> > process so that any DT file that doesn't match the documented schema
> > will get flagged, and that the schema files will be human readable and
> > will double as documentation.
> >
> > There is not yet any process for binding maintainership. We talked about
> > a few ideas, but they really need to be hashed out here on the mailing
> > list. A couple of the questions:
> >
> > - How are bindings allowed to be merged? Through subsystem trees, or
> > only through the bindings tree?
> > - Through the bindings tree is more work but it will provide more
> > control.
> > - Through subsystem trees means drivers and bindings get merged
> > together.
> > - If we have a schema tool that reports errors on missing or
> > unapproved schema, then spliting the driver from the binding won't
> > matter that much.
> > - Do we need to differentiate between 'staging' and 'stable' bindings?
> > - What is the schedule for splitting the bindings and .dts files out of
> > the kernel?
> > - Ian Campbell is maintaining a DT bindings and .dts mirror tree which
> > should eventually become the 'master' for merging DT bindings.
>
> I remember getting to a conclusion that:
> - bindings should enter staging state after being introduced,
> - from time to time a binding review meeting should take place (on IRC
> possibly) and discuss which of introduced staging bindings are ready to
> enter stable state,
> - in stable state such binding would be considered an ABI.
>
> From remaining questions I remember:
> - How should we mark bindings as staging and stable? (i.e.
> documentation/schema files in different folders or something else?)
I think having a different folder for staged bindings is going to be the
most managable. The other option would be a tag in the file indicating
that it is stable, but I suspect that will require more work in the long
run. At least with a staging directory you can tell at a glance which
bindings are in draft form. Any bindings used from the staging directory
should trigger a warning from DTC, but we probably want to consolidate
all 'staging' binding warnings into a single message so that dtc output
doesn't get too chatty.
Once a binding is moved into the stable directory, only backwards
compatible changes should be allowed.
> - Should we also split parts of dts/dtsi using staging bindings from
> those using stable ones? (This could mean including stable dts/dtsi file
> from inside unstable one, which would allow having a stable dts with basic
> functionality that could be extended over the time after validating
> staging bindings used in unstable part.)
Good question. Need to try it out and see how it works from a
maintainership standpoint.
> As for my part, I'm now looking into existing infrastructure inside of dtc
> to get some hints that would allow me to design the initial schema syntax
> in a most dtc-friendly way. Give me a bit more time and I will then write
> down everything I have and post to the ML to start a discussion.
I think for any given schema, it should probably have the following
elements or sections:
1) A set of properties/characteristics to match against.
- The most likely case here will be entries in the compatible
property, but there are a few device_type and name matches that will
need to be supported
2) a set of required/optional properties and child nodes
3) a set of 'inhereted' schema.
- for example, the vast majority of device bindings will want to
inherit from the common 'reg' and 'interrupts' definitions.
- Another example: a gpio controller will want to inherit the generic
gpio controller schema.
I wouldn't get to hung up about trying to reuse the same file format as
the .dts files. Most of the bindings are going to be very flat in
nature, and schemas need to be expresive in a different way from the
.dts files themselves.
We'll want to have an easy way to define trivial bindings. There are a
lot of devices that only define a new 'compatible' string, but they
don't actually require anything more than reg and interrupts. Can we
maybe have a schema file that lists trivial bindings?
There will need to be a way to describe cross-schema interactions. For
example, consider the gpio-leds binding and the gpio controller binding.
The gpio-controller node needs to have inherited the gpio controller
schema. Similarly, the gpio-leds schema will need the gpio-consumer
binding, but it will also need to verify that the gpio-controller
conforms to the gpio controller schema.
Here's a really rough draft at a few schema files. This is /not/ a
really good syntax yet. One problematic part is figuring out how to
dereference values from other nodes.
core-reg.dtschema:
required: reg = <[%address% %size%]*> // '*' indicates one or more tuples
%address%: size = parent.#address-cells * 4;
%size%: size = parent.#size-cells * 4;
core-interrupt.dtschema:
required: interrupts = <[%intspec%]*>
%intspec%: size = %interrupt-parent%:#interrupt-cells;
optional: interrupt-parent = <%interrupt-phandle%> // need a way to search up the tree
core-interrupt-controller.dtschema:
required: interrupt-controller;
required: #interrupt-cells = <num>;
%intspec%: size = interrupt-parent:#interrupt-cells;
arm-gic.dtschema:
match: compatible = ".*arm,{cortex-{a15,a9,a7},arm11mp}-gic.*"
inherit-required: core-reg;
inherit-required: core-interrupt-controller;
inherit-optional: core-interrupt;
required: #interrupt-cells = <3>;
g.
On Wed, 24 Jul 2013, Grant Likely wrote:
> I wouldn't get to hung up about trying to reuse the same file format as
> the .dts files. Most of the bindings are going to be very flat in
> nature, and schemas need to be expresive in a different way from the
> .dts files themselves.
>
> We'll want to have an easy way to define trivial bindings. There are a
> lot of devices that only define a new 'compatible' string, but they
> don't actually require anything more than reg and interrupts. Can we
> maybe have a schema file that lists trivial bindings?
It sounds like a log of the bindings will really consist of something similar to
"take this other binding definition and change these values"
I think that figuring out a good way to handle this (and make sure that when
people change one binding, they can know what other bindings they are changing
through the dependancies) is at least as important as figuring out the formal
schemas and multiple inheritance issues.
David Lang
On 07/24/2013 07:11 AM, Grant Likely wrote:
> On Sat, 20 Jul 2013 15:49:21 +0200, Tomasz Figa <[email protected]> wrote:
...
>> From remaining questions I remember:
>> - How should we mark bindings as staging and stable? (i.e.
>> documentation/schema files in different folders or something else?)
>
> I think having a different folder for staged bindings is going to be the
> most managable. The other option would be a tag in the file indicating
> that it is stable, but I suspect that will require more work in the long
> run. At least with a staging directory you can tell at a glance which
> bindings are in draft form. Any bindings used from the staging directory
> should trigger a warning from DTC, but we probably want to consolidate
> all 'staging' binding warnings into a single message so that dtc output
> doesn't get too chatty.
>
> Once a binding is moved into the stable directory, only backwards
> compatible changes should be allowed.
Is the following scenario useful to consider
* A minimal DT binding is initially created to represent the most basic
features of a device. This graduates quickly to being marked stable.
* The full feature-set of the device needs to be exposed later. This
involves making backwards-compatible changes to the binding. Those
changes may initially be very tentative, since they're more complex.
This implies to me that we either need to mark specific properties or
parts of a binding as stable, and part as tentative. Does using separate
directories for this make sense, or should we put markup into the
binding definition itself?
I suppose we could consider bindings/tentative/foo/bar.txt as
automatically inheriting-from/extending bindings/stable/foo/bar.txt.