Hi all,
Currently we have no generic/standard mechanisms for describing the
relationship between PCI root complexes and other components which may be
required to make them usable, specifically IOMMUs and MSI controllers.
There is an existing binding for IOMMUs, and there is a de-facto standard for
referring to MSI controllers, but the generic portion of these can only
describe a relationship with a root complex as opposed to a device under a root
complex. This falls apart where IOMMUs and MSI controllers may distinguish
individual devices based on non-probeable information.
This series adds a generic "glue" bindings for describing the relationship
between root complexes, IOMMUs, and MSI controllers. The existing de-facto
binding for MSI controllers is formalised, along with a (backwards compatible)
extension necessary for describing contemporary MSI controllers which make use
of (non-probeable) sideband data.
Thanks,
Mark.
Mark Rutland (3):
Docs: dt: add generic MSI bindings
Docs: dt: Add PCI MSI map bindings
Docs: dt: add PCI IOMMU map bindings
.../bindings/interrupt-controller/msi.txt | 135 +++++++++++++
.../devicetree/bindings/pci/pci-iommu.txt | 171 ++++++++++++++++
Documentation/devicetree/bindings/pci/pci-msi.txt | 220 +++++++++++++++++++++
3 files changed, 526 insertions(+)
create mode 100644 Documentation/devicetree/bindings/interrupt-controller/msi.txt
create mode 100644 Documentation/devicetree/bindings/pci/pci-iommu.txt
create mode 100644 Documentation/devicetree/bindings/pci/pci-msi.txt
--
1.9.1
Currently msi-parent is used in a couple of drivers despite being fairly
underspecified. This patch adds a generic binding for MSIs (including
the existing msi-parent property) enabling the description of platform
devices capable of using MSIs.
While MSIs are primarily distinguished by doorbell and payload, some MSI
controllers (e.g. the GICv3 ITS) also use side-band information
accompanying the write to identify the master which originated the MSI,
to allow for sandboxing. This sideband information is non-probeable and
needs to be described in the DT. Other MSI controllers may have
additional configuration details which need to be described per-master.
This patch adds a generic msi-parent binding document, extending the
de-facto standard with a new (optional) #msi-cells which can be used to
express any per-master configuration and/or sideband data. This is
sufficient to describe non-hotpluggable devices.
For busses where sideband data may be derived from some bus-specific
master ID scheme, other properties will be required to describe the
mapping.
Signed-off-by: Mark Rutland <[email protected]>
---
.../bindings/interrupt-controller/msi.txt | 135 +++++++++++++++++++++
1 file changed, 135 insertions(+)
create mode 100644 Documentation/devicetree/bindings/interrupt-controller/msi.txt
diff --git a/Documentation/devicetree/bindings/interrupt-controller/msi.txt b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
new file mode 100644
index 0000000..c60c034
--- /dev/null
+++ b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
@@ -0,0 +1,135 @@
+This document describes the generic device tree binding for MSI controllers and
+their master(s).
+
+Message Signaled Interrupts (MSIs) are a class of interrupts generated by a
+write to an MMIO address.
+
+MSIs were originally specified by PCI (and are used with PCIe), but may also be
+used with other busses, and hence a mechanism is required to relate devices on
+those busses to the MSI controllers which they are capable of using,
+potentially including additional information.
+
+MSIs are distinguished by some combination of:
+
+- The doorbell (the MMIO address written to).
+
+ Devices may be configured by software to write to arbitrary doorbells which
+ they can address. An MSI controller may feature a number of doorbells.
+
+- The payload (the value written to the doorbell).
+
+ Devices may be configured to write an arbitrary payload chosen by software.
+ MSI controllers may have restrictions on permitted payloads.
+
+- Sideband information accompanying the write.
+
+ Typically this is neither configurable nor probeable, and depends on the path
+ taken through the memory system (i.e. it is a property of the combination of
+ MSI controller and device rather than a property of either in isolation).
+
+
+MSI controllers:
+================
+
+An MSI controller signals interrupts to a CPU when a write is made to an MMIO
+address by some master. An MSI controller may feature a number of doorbells.
+
+Required properties:
+--------------------
+
+- msi-controller: Identifies the node as an MSI controller.
+
+Optional properties:
+--------------------
+
+- #msi-cells: The number of cells in an msi-specifier, required if not zero.
+
+ Typically this will encode information related to sideband data, and will
+ not encode doorbells or payloads as these can be configured dynamically.
+
+ The meaning of the msi-specifier is defined by the device tree binding of
+ the specific MSI controller.
+
+
+MSI clients
+===========
+
+MSI clients are devices which generate MSIs. For each MSI they wish to
+generate, the doorbell and payload may be configured, though sideband
+information may not be configurable.
+
+Required properties:
+--------------------
+
+- msi-parent: A list of phandle + msi-specifier pairs, one for each MSI
+ controller which the device is capable of using.
+
+ This property is unordered, and MSIs may be allocated from any combination of
+ MSI controllers listed in the msi-parent property.
+
+ If a device has restrictions on the allocation of MSIs, these restrictions
+ must be described with additional properties.
+
+ When #msi-cells is non-zero, busses with an msi-parent will require
+ additional properties to describe the relationship between devices on the bus
+ and the set of MSIs they can potentially generate.
+
+
+Example
+=======
+
+/ {
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ msi_a: msi-controller@a {
+ reg = <0xa 0xf00>;
+ compatible = "vendor-a,some-controller";
+ msi-controller;
+ /* No sideband data, so #msi-cells omitted */
+ };
+
+ msi_b: msi-controller@b {
+ reg = <0xb 0xf00>;
+ compatible = "vendor-b,another-controller";
+ msi-controller;
+ /* Each device has some unique ID */
+ #msi-cells = <1>;
+ };
+
+ msi_c: msi-controller@c {
+ reg = <0xb 0xf00>;
+ compatible = "vendor-b,another-controller";
+ msi-controller;
+ /* Each device has some unique ID */
+ #msi-cells = <1>;
+ };
+
+ dev@0 {
+ reg = <0x0 0xf00>;
+ compatible = "vendor-c,some-device";
+
+ /* Can only generate MSIs to msi_a */
+ msi-parent = <&msi_a>;
+ };
+
+ dev@1 {
+ reg = <0x1 0xf00>;
+ compatible = "vendor-c,some-device";
+
+ /*
+ * Can generate MSIs to either A or B.
+ */
+ msi-parent = <&msi_a>, <&msi_b 0x17>;
+ };
+
+ dev@2 {
+ reg = <0x2 0xf00>;
+ compatible = "vendor-c,some-device";
+ /*
+ * Has different IDs at each MSI controller.
+ * Can generate MSIs to all of the MSI controllers.
+ */
+ msi-parent = <&msi_a>, <&msi_b 0x17>, <&msi_c 0x53>;
+ };
+};
--
1.9.1
Currently msi-parent is used by a few bindings to describe the
relationship between a PCI root complex and a single MSI controller, but
this property does not have a generic binding document.
Additionally, msi-parent is insufficient to describe more complex
relationships between MSI controllers and devices under a root complex,
where devices may be able to target multiple MSI controllers, or where
MSI controllers use (non-probeable) sideband information to distinguish
devices.
This patch adds a generic binding for mapping PCI devices to MSI
controllers. This document covers msi-parent, and a new msi-map property
(specific to PCI*) which may be used to map devices (identified by their
Requester ID) to sideband data for each MSI controller that they may
target.
Signed-off-by: Mark Rutland <[email protected]>
---
Documentation/devicetree/bindings/pci/pci-msi.txt | 220 ++++++++++++++++++++++
1 file changed, 220 insertions(+)
create mode 100644 Documentation/devicetree/bindings/pci/pci-msi.txt
diff --git a/Documentation/devicetree/bindings/pci/pci-msi.txt b/Documentation/devicetree/bindings/pci/pci-msi.txt
new file mode 100644
index 0000000..9b3cc81
--- /dev/null
+++ b/Documentation/devicetree/bindings/pci/pci-msi.txt
@@ -0,0 +1,220 @@
+This document describes the generic device tree binding for describing the
+relationship between PCI devices and MSI controllers.
+
+Each PCI device under a root complex is uniquely identified by its Requester ID
+(AKA RID). A Requester ID is a triplet of a Bus number, Device number, and
+Function number.
+
+For the purpose of this document, when treated as a numeric value, a RID is
+formatted such that:
+
+* Bits [15:8] are the Bus number.
+* Bits [7:3] are the Device number.
+* Bits [2:0] are the Function number.
+* Any other bits required for padding must be zero.
+
+MSIs may be distinguished in part through the use of sideband data accompanying
+writes. In the case of PCI devices, this sideband data may be derived from the
+Requester ID. A mechanism is required to associate a device with both the MSI
+controllers it can address, and the sideband data that will be associated with
+its writes to those controllers.
+
+For generic MSI bindings, see
+Documentation/devicetree/bindings/interrupt-controller/msi.txt.
+
+
+PCI root complex
+================
+
+Optional properties
+-------------------
+
+- msi-map: Maps a Requester ID to an MSI controller and associated
+ msi-specifier data. The property is an arbitrary number of tuples of
+ (rid-base,msi-controller,msi-base,length), where:
+
+ * rid-base is a single cell describing the first RID matched by the entry.
+
+ * msi-controller is a single phandle to an MSI controller
+
+ * msi-base is an msi-specifier describing the msi-specifier produced for the
+ first RID matched by the entry.
+
+ * length is a single cell describing how many consecutive RIDs are matched
+ following the rid-base.
+
+ Any RID r in the interval [rid-base, rid-base + length) is associated with
+ the listed msi-controller, with the msi-specifier (r - rid-base + msi-base).
+
+- msi-map-mask: A mask to be applied to each Requester ID prior to being mapped
+ to an msi-specifier per the msi-map property.
+
+- msi-parent: Describes the MSI parent of the root complex itself. Where
+ the root complex and MSI controller do not pass sideband data with MSI
+ writes, this property may be used to describe the MSI controller(s)
+ used by PCI devices under the root complex, if defined as such in the
+ binding for the root complex.
+
+
+Example (1)
+===========
+
+/ {
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ msi: msi-controller@a {
+ reg = <0xa 0x1>;
+ compatible = "vendor,some-controller";
+ msi-controller;
+ #msi-cells = <1>;
+ };
+
+ pci: pci@f {
+ reg = <0xf 0x1>;
+ compatible = "vendor,pcie-root-complex";
+ device_type = "pci";
+
+ /*
+ * The sideband data provided to the MSI controller is
+ * the RID, identity-mapped.
+ */
+ msi-map = <0x0 &msi_a 0x0 0x10000>,
+ };
+};
+
+
+Example (2)
+===========
+
+/ {
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ msi: msi-controller@a {
+ reg = <0xa 0x1>;
+ compatible = "vendor,some-controller";
+ msi-controller;
+ #msi-cells = <1>;
+ };
+
+ pci: pci@f {
+ reg = <0xf 0x1>;
+ compatible = "vendor,pcie-root-complex";
+ device_type = "pci";
+
+ /*
+ * The sideband data provided to the MSI controller is
+ * the RID, masked to only the device and function bits.
+ */
+ msi-map = <0x0 &msi_a 0x0 0x100>,
+ msi-map-mask = <0xff>
+ };
+};
+
+
+Example (3)
+===========
+
+/ {
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ msi: msi-controller@a {
+ reg = <0xa 0x1>;
+ compatible = "vendor,some-controller";
+ msi-controller;
+ #msi-cells = <1>;
+ };
+
+ pci: pci@f {
+ reg = <0xf 0x1>;
+ compatible = "vendor,pcie-root-complex";
+ device_type = "pci";
+
+ /*
+ * The sideband data provided to the MSI controller is
+ * the RID, but the high bit of the bus number is
+ * ignored.
+ */
+ msi-map = <0x0000 &msi 0x0000 0x8000>,
+ <0x8000 &msi 0x0000 0x8000>;
+ };
+};
+
+
+Example (4)
+===========
+
+/ {
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ msi: msi-controller@a {
+ reg = <0xa 0x1>;
+ compatible = "vendor,some-controller";
+ msi-controller;
+ #msi-cells = <1>;
+ };
+
+ pci: pci@f {
+ reg = <0xf 0x1>;
+ compatible = "vendor,pcie-root-complex";
+ device_type = "pci";
+
+ /*
+ * The sideband data provided to the MSI controller is
+ * the RID, but the high bit of the bus number is
+ * negated.
+ */
+ msi-map = <0x0000 &msi 0x8000 0x8000>,
+ <0x8000 &msi 0x0000 0x8000>;
+ };
+};
+
+
+Example (5)
+===========
+
+/ {
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ msi_a: msi-controller@a {
+ reg = <0xa 0x1>;
+ compatible = "vendor,some-controller";
+ msi-controller;
+ #msi-cells = <1>;
+ };
+
+ msi_b: msi-controller@b {
+ reg = <0xb 0x1>;
+ compatible = "vendor,some-controller";
+ msi-controller;
+ #msi-cells = <1>;
+ };
+
+ msi_c: msi-controller@c {
+ reg = <0xc 0x1>;
+ compatible = "vendor,some-controller";
+ msi-controller;
+ #msi-cells = <1>;
+ };
+
+ pci: pci@c {
+ reg = <0xf 0x1>;
+ compatible = "vendor,pcie-root-complex";
+ device_type = "pci";
+
+ /*
+ * The sideband data provided to MSI controller a is the
+ * RID, but the high bit of the bus number is negated.
+ * The sideband data provided to MSI controller b is the
+ * RID, identity-mapped.
+ * MSI controller c is not addressable.
+ */
+ msi-map = <0x0000 &msi_a 0x8000 0x08000>,
+ <0x8000 &msi_a 0x0000 0x08000>,
+ <0x0000 &msi_b 0x0000 0x10000>;
+ };
+};
--
1.9.1
The existing IOMMU bindings are able to specify the relationship between
masters and IOMMUs, but they are insufficient for describing the general
case of hotpluggable busses such as PCI where the set of masters is not
known until runtime, and the relationship between masters and IOMMUs is
a property of the integration of the system.
This patch adds a generic binding for mapping PCI devices to IOMMUs,
using a new iommu-map property (specific to PCI*) which may be used to
map devices (identified by their Requester ID) to sideband data for the
IOMMU which they master through.
Signed-off-by: Mark Rutland <[email protected]>
---
.../devicetree/bindings/pci/pci-iommu.txt | 171 +++++++++++++++++++++
1 file changed, 171 insertions(+)
create mode 100644 Documentation/devicetree/bindings/pci/pci-iommu.txt
diff --git a/Documentation/devicetree/bindings/pci/pci-iommu.txt b/Documentation/devicetree/bindings/pci/pci-iommu.txt
new file mode 100644
index 0000000..56c8296
--- /dev/null
+++ b/Documentation/devicetree/bindings/pci/pci-iommu.txt
@@ -0,0 +1,171 @@
+This document describes the generic device tree binding for describing the
+relationship between PCI(e) devices and IOMMU(s).
+
+Each PCI(e) device under a root complex is uniquely identified by its Requester
+ID (AKA RID). A Requester ID is a triplet of a Bus number, Device number, and
+Function number.
+
+For the purpose of this document, when treated as a numeric value, a RID is
+formatted such that:
+
+* Bits [15:8] are the Bus number.
+* Bits [7:3] are the Device number.
+* Bits [2:0] are the Function number.
+* Any other bits required for padding must be zero.
+
+IOMMUs may distinguish PCI devices through sideband data derived from the
+Requester ID. While a given PCI device can only master through one IOMMU, a
+root complex may split masters across a set of IOMMUs (e.g. with one IOMMU per
+bus).
+
+The generic 'iommus' property is insufficient to describe this relationship,
+and a mechanism is required to map from a PCI device to its IOMMU and sideband
+data.
+
+For generic IOMMU bindings, see
+Documentation/devicetree/bindings/iommu/iommu.txt.
+
+
+PCI root complex
+================
+
+Optional properties
+-------------------
+
+- iommu-map: Maps a Requester ID to an IOMMU and associated iommu-specifier
+ data.
+
+ The property is an arbitrary number of tuples of
+ (rid-base,iommu,iommu-base,length).
+
+ Any RID r in the interval [rid-base, rid-base + length) is associated with
+ the listed IOMMU, with the iommu-specifier (r - rid-base + iommu-base).
+
+- iommu-map-mask: A mask to be applied to each Requester ID prior to being
+ mapped to an iommu-specifier per the iommu-map property.
+
+
+Example (1)
+===========
+
+/ {
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ iommu: iommu@a {
+ reg = <0xa 0x1>;
+ compatible = "vendor,some-iommu";
+ #iommu-cells = <1>;
+ };
+
+ pci: pci@f {
+ reg = <0xf 0x1>;
+ compatible = "vendor,pcie-root-complex";
+ device_type = "pci";
+
+ /*
+ * The sideband data provided to the IOMMU is the RID,
+ * identity-mapped.
+ */
+ iommu-map = <0x0 &iommu 0x0 0x10000>;
+ };
+};
+
+
+Example (2)
+===========
+
+/ {
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ iommu: iommu@a {
+ reg = <0xa 0x1>;
+ compatible = "vendor,some-iommu";
+ #iommu-cells = <1>;
+ };
+
+ pci: pci@f {
+ reg = <0xf 0x1>;
+ compatible = "vendor,pcie-root-complex";
+ device_type = "pci";
+
+ /*
+ * The sideband data provided to the IOMMU is the RID with the
+ * function bits masked out.
+ */
+ iommu-map = <0x0 &iommu 0x0 0x10000>;
+ iommu-map-mask = <0xfff8>;
+ };
+};
+
+
+Example (3)
+===========
+
+/ {
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ iommu: iommu@a {
+ reg = <0xa 0x1>;
+ compatible = "vendor,some-iommu";
+ #iommu-cells = <1>;
+ };
+
+ pci: pci@f {
+ reg = <0xf 0x1>;
+ compatible = "vendor,pcie-root-complex";
+ device_type = "pci";
+
+ /*
+ * The sideband data provided to the IOMMU is the RID,
+ * but the high bits of the bus number are flipped.
+ */
+ iommu-map = <0x0000 &iommu 0x8000 0x8000>,
+ <0x8000 &iommu 0x0000 0x8000>;
+ };
+};
+
+
+Example (4)
+===========
+
+/ {
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ iommu_a: iommu@a {
+ reg = <0xa 0x1>;
+ compatible = "vendor,some-iommu";
+ #iommu-cells = <1>;
+ };
+
+ iommu_b: iommu@b {
+ reg = <0xb 0x1>;
+ compatible = "vendor,some-iommu";
+ #iommu-cells = <1>;
+ };
+
+ iommu_c: iommu@c {
+ reg = <0xc 0x1>;
+ compatible = "vendor,some-iommu";
+ #iommu-cells = <1>;
+ };
+
+ pci: pci@f {
+ reg = <0xf 0x1>;
+ compatible = "vendor,pcie-root-complex";
+ device_type = "pci";
+
+ /*
+ * Devices with bus number 0-127 are mastered via IOMMU
+ * a, with sideband data being RID[14:0].
+ * Devices with bus number 128-255 are mastered via
+ * IOMMU b, with sideband data being RID[14:0].
+ * No devices master via IOMMU c.
+ */
+ iommu-map = <0x0000 &iommu_a 0x0000 0x8000>,
+ <0x8000 &iommu_b 0x0000 0x8000>;
+ };
+};
--
1.9.1
On 07/23/2015 09:52 AM, Mark Rutland wrote:
[...]
> +MSI clients
> +===========
> +
> +MSI clients are devices which generate MSIs. For each MSI they wish to
> +generate, the doorbell and payload may be configured, though sideband
> +information may not be configurable.
> +
> +Required properties:
> +--------------------
> +
> +- msi-parent: A list of phandle + msi-specifier pairs, one for each MSI
> + controller which the device is capable of using.
> +
We say here that "msi-parent" consists of pairs ...
> + This property is unordered, and MSIs may be allocated from any combination of
> + MSI controllers listed in the msi-parent property.
> +
> + If a device has restrictions on the allocation of MSIs, these restrictions
> + must be described with additional properties.
> +
> + When #msi-cells is non-zero, busses with an msi-parent will require
> + additional properties to describe the relationship between devices on the bus
> + and the set of MSIs they can potentially generate.
> +
> +
> +Example
> +=======
> +
> +/ {
> + #address-cells = <1>;
> + #size-cells = <1>;
> +
> + msi_a: msi-controller@a {
> + reg = <0xa 0xf00>;
> + compatible = "vendor-a,some-controller";
> + msi-controller;
> + /* No sideband data, so #msi-cells omitted */
> + };
> +
> + msi_b: msi-controller@b {
> + reg = <0xb 0xf00>;
> + compatible = "vendor-b,another-controller";
> + msi-controller;
> + /* Each device has some unique ID */
> + #msi-cells = <1>;
> + };
> +
> + msi_c: msi-controller@c {
> + reg = <0xb 0xf00>;
> + compatible = "vendor-b,another-controller";
> + msi-controller;
> + /* Each device has some unique ID */
> + #msi-cells = <1>;
> + };
> +
> + dev@0 {
> + reg = <0x0 0xf00>;
> + compatible = "vendor-c,some-device";
> +
> + /* Can only generate MSIs to msi_a */
> + msi-parent = <&msi_a>;
My device-tree syntax foo is a little rusty, but doesn't "msi-parent"
need a pair of elements? This has only the phandle.
> + };
> +
> + dev@1 {
> + reg = <0x1 0xf00>;
> + compatible = "vendor-c,some-device";
> +
> + /*
> + * Can generate MSIs to either A or B.
> + */
> + msi-parent = <&msi_a>, <&msi_b 0x17>;
... same here, ...
> + };
> +
> + dev@2 {
> + reg = <0x2 0xf00>;
> + compatible = "vendor-c,some-device";
> + /*
> + * Has different IDs at each MSI controller.
> + * Can generate MSIs to all of the MSI controllers.
> + */
> + msi-parent = <&msi_a>, <&msi_b 0x17>, <&msi_c 0x53>;
... and here
> + };
> +};
>
On Thu, 23 Jul 2015 19:26:11 +0100
David Daney <[email protected]> wrote:
> On 07/23/2015 09:52 AM, Mark Rutland wrote:
> [...]
> > +MSI clients
> > +===========
> > +
> > +MSI clients are devices which generate MSIs. For each MSI they wish to
> > +generate, the doorbell and payload may be configured, though sideband
> > +information may not be configurable.
> > +
> > +Required properties:
> > +--------------------
> > +
> > +- msi-parent: A list of phandle + msi-specifier pairs, one for each MSI
> > + controller which the device is capable of using.
> > +
>
> We say here that "msi-parent" consists of pairs ...
>
> > + This property is unordered, and MSIs may be allocated from any combination of
> > + MSI controllers listed in the msi-parent property.
> > +
> > + If a device has restrictions on the allocation of MSIs, these restrictions
> > + must be described with additional properties.
> > +
> > + When #msi-cells is non-zero, busses with an msi-parent will require
> > + additional properties to describe the relationship between devices on the bus
> > + and the set of MSIs they can potentially generate.
> > +
> > +
> > +Example
> > +=======
> > +
> > +/ {
> > + #address-cells = <1>;
> > + #size-cells = <1>;
> > +
> > + msi_a: msi-controller@a {
> > + reg = <0xa 0xf00>;
> > + compatible = "vendor-a,some-controller";
> > + msi-controller;
> > + /* No sideband data, so #msi-cells omitted */
> > + };
> > +
> > + msi_b: msi-controller@b {
> > + reg = <0xb 0xf00>;
> > + compatible = "vendor-b,another-controller";
> > + msi-controller;
> > + /* Each device has some unique ID */
> > + #msi-cells = <1>;
> > + };
> > +
> > + msi_c: msi-controller@c {
> > + reg = <0xb 0xf00>;
> > + compatible = "vendor-b,another-controller";
> > + msi-controller;
> > + /* Each device has some unique ID */
> > + #msi-cells = <1>;
> > + };
> > +
> > + dev@0 {
> > + reg = <0x0 0xf00>;
> > + compatible = "vendor-c,some-device";
> > +
> > + /* Can only generate MSIs to msi_a */
> > + msi-parent = <&msi_a>;
>
>
> My device-tree syntax foo is a little rusty, but doesn't "msi-parent"
> need a pair of elements? This has only the phandle.
This is a pair in the sense of (phandle, msi-specifier).
msi_a doesn't have a #msi-cells, so its msi-specifier is of size 0. In
that case, the pair is (phandle, empty-set).
You could also have something like:
msi_d: msi-controller@d {
#msi-cells = <2>;
};
dev@f {
msi-parent = <&msi_d 0x1 0x2>;
};
and msi-parent in this case would still be a pair, with an
msi-specifier of size 2.
Hope this helps,
M.
--
Without deviation from the norm, progress is not possible.
Hi Mark,
This looks sane, and lets me describe the thing I have on my desk, so
I'm happy. I have a couple of general thoughts below, but I don't intend
that they should stand in the way of this proposal as-is.
On 23/07/15 17:52, Mark Rutland wrote:
> The existing IOMMU bindings are able to specify the relationship between
> masters and IOMMUs, but they are insufficient for describing the general
> case of hotpluggable busses such as PCI where the set of masters is not
> known until runtime, and the relationship between masters and IOMMUs is
> a property of the integration of the system.
>
> This patch adds a generic binding for mapping PCI devices to IOMMUs,
> using a new iommu-map property (specific to PCI*) which may be used to
> map devices (identified by their Requester ID) to sideband data for the
> IOMMU which they master through.
>
> Signed-off-by: Mark Rutland <[email protected]>
> ---
> .../devicetree/bindings/pci/pci-iommu.txt | 171 +++++++++++++++++++++
> 1 file changed, 171 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/pci/pci-iommu.txt
>
> diff --git a/Documentation/devicetree/bindings/pci/pci-iommu.txt b/Documentation/devicetree/bindings/pci/pci-iommu.txt
> new file mode 100644
> index 0000000..56c8296
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/pci/pci-iommu.txt
> @@ -0,0 +1,171 @@
> +This document describes the generic device tree binding for describing the
> +relationship between PCI(e) devices and IOMMU(s).
> +
> +Each PCI(e) device under a root complex is uniquely identified by its Requester
> +ID (AKA RID). A Requester ID is a triplet of a Bus number, Device number, and
> +Function number.
> +
> +For the purpose of this document, when treated as a numeric value, a RID is
> +formatted such that:
> +
> +* Bits [15:8] are the Bus number.
> +* Bits [7:3] are the Device number.
> +* Bits [2:0] are the Function number.
> +* Any other bits required for padding must be zero.
> +
> +IOMMUs may distinguish PCI devices through sideband data derived from the
> +Requester ID. While a given PCI device can only master through one IOMMU, a
> +root complex may split masters across a set of IOMMUs (e.g. with one IOMMU per
> +bus).
> +
> +The generic 'iommus' property is insufficient to describe this relationship,
> +and a mechanism is required to map from a PCI device to its IOMMU and sideband
> +data.
> +
> +For generic IOMMU bindings, see
> +Documentation/devicetree/bindings/iommu/iommu.txt.
> +
> +
> +PCI root complex
> +================
> +
> +Optional properties
> +-------------------
> +
> +- iommu-map: Maps a Requester ID to an IOMMU and associated iommu-specifier
> + data.
> +
> + The property is an arbitrary number of tuples of
> + (rid-base,iommu,iommu-base,length).
> +
> + Any RID r in the interval [rid-base, rid-base + length) is associated with
> + the listed IOMMU, with the iommu-specifier (r - rid-base + iommu-base).
Can we take as a guarantee that the system cannot present any ID at a
given IOMMU that is not represented in an appropriate output range (in
the sense that we may do things that could blow up horribly if spurious
IDs appeared)?
Furthermore, would representing one-to-many mappings by having multiple
matches for a given RID be legal? In the general case it's certainly
feasible for the IOMMU to see different IDs for e.g. reads vs. writes,
where the system munges extra bus lines into the sideband signals -
whether anyone would actually integrate a PCI host controller that way
is another matter, so I don't think it's something worth really worrying
about without a definite need.
> +
> +- iommu-map-mask: A mask to be applied to each Requester ID prior to being
> + mapped to an iommu-specifier per the iommu-map property.
Am I right to assume a mask of 0 would be a valid way to represent
"everything" (and if so, should rid-base and length just be ignored, or
mandated to be 0 and 1 respectively)? It looks a bit off at first
glance, but it does neatly address a genuine use-case.
> +
> +
> +Example (1)
> +===========
> +
> +/ {
> + #address-cells = <1>;
> + #size-cells = <1>;
> +
> + iommu: iommu@a {
> + reg = <0xa 0x1>;
> + compatible = "vendor,some-iommu";
> + #iommu-cells = <1>;
Troll question; what do we do when #iommu-cells > 1, where the IOMMU is
expecting some extra data associated with each ID (say, memory attributes)?
[ I'm pretty sure the answer here should be "define some additional
binding if and when anyone actually cares" ;) ]
Robin.
> + };
> +
> + pci: pci@f {
> + reg = <0xf 0x1>;
> + compatible = "vendor,pcie-root-complex";
> + device_type = "pci";
> +
> + /*
> + * The sideband data provided to the IOMMU is the RID,
> + * identity-mapped.
> + */
> + iommu-map = <0x0 &iommu 0x0 0x10000>;
> + };
> +};
Hi,
> This looks sane, and lets me describe the thing I have on my desk, so
> I'm happy. I have a couple of general thoughts below, but I don't intend
> that they should stand in the way of this proposal as-is.
Good to hear that this doesn't fall apart at the sight of a real system!
> On 23/07/15 17:52, Mark Rutland wrote:
> > The existing IOMMU bindings are able to specify the relationship between
> > masters and IOMMUs, but they are insufficient for describing the general
> > case of hotpluggable busses such as PCI where the set of masters is not
> > known until runtime, and the relationship between masters and IOMMUs is
> > a property of the integration of the system.
> >
> > This patch adds a generic binding for mapping PCI devices to IOMMUs,
> > using a new iommu-map property (specific to PCI*) which may be used to
> > map devices (identified by their Requester ID) to sideband data for the
> > IOMMU which they master through.
> >
> > Signed-off-by: Mark Rutland <[email protected]>
> > ---
> > .../devicetree/bindings/pci/pci-iommu.txt | 171 +++++++++++++++++++++
> > 1 file changed, 171 insertions(+)
> > create mode 100644 Documentation/devicetree/bindings/pci/pci-iommu.txt
> >
> > diff --git a/Documentation/devicetree/bindings/pci/pci-iommu.txt b/Documentation/devicetree/bindings/pci/pci-iommu.txt
> > new file mode 100644
> > index 0000000..56c8296
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/pci/pci-iommu.txt
> > @@ -0,0 +1,171 @@
> > +This document describes the generic device tree binding for describing the
> > +relationship between PCI(e) devices and IOMMU(s).
> > +
> > +Each PCI(e) device under a root complex is uniquely identified by its Requester
> > +ID (AKA RID). A Requester ID is a triplet of a Bus number, Device number, and
> > +Function number.
> > +
> > +For the purpose of this document, when treated as a numeric value, a RID is
> > +formatted such that:
> > +
> > +* Bits [15:8] are the Bus number.
> > +* Bits [7:3] are the Device number.
> > +* Bits [2:0] are the Function number.
> > +* Any other bits required for padding must be zero.
> > +
> > +IOMMUs may distinguish PCI devices through sideband data derived from the
> > +Requester ID. While a given PCI device can only master through one IOMMU, a
> > +root complex may split masters across a set of IOMMUs (e.g. with one IOMMU per
> > +bus).
> > +
> > +The generic 'iommus' property is insufficient to describe this relationship,
> > +and a mechanism is required to map from a PCI device to its IOMMU and sideband
> > +data.
> > +
> > +For generic IOMMU bindings, see
> > +Documentation/devicetree/bindings/iommu/iommu.txt.
> > +
> > +
> > +PCI root complex
> > +================
> > +
> > +Optional properties
> > +-------------------
> > +
> > +- iommu-map: Maps a Requester ID to an IOMMU and associated iommu-specifier
> > + data.
> > +
> > + The property is an arbitrary number of tuples of
> > + (rid-base,iommu,iommu-base,length).
> > +
> > + Any RID r in the interval [rid-base, rid-base + length) is associated with
> > + the listed IOMMU, with the iommu-specifier (r - rid-base + iommu-base).
>
> Can we take as a guarantee that the system cannot present any ID at a
> given IOMMU that is not represented in an appropriate output range (in
> the sense that we may do things that could blow up horribly if spurious
> IDs appeared)?
I would expect that for the root complex, the iommu-map should cover all
possible RIDs (and hence we would know their output IDs). In the case
that a possible RID was not translated by the property, I would hope
that we could either detect this at parse time, or prevent probing of
the device when it appeared.
The root complex could share IOMMUs with other masters, so in general
iommu-map alone would not be sufficient to detect all possible IDs.
> Furthermore, would representing one-to-many mappings by having multiple
> matches for a given RID be legal? In the general case it's certainly
> feasible for the IOMMU to see different IDs for e.g. reads vs. writes,
> where the system munges extra bus lines into the sideband signals -
> whether anyone would actually integrate a PCI host controller that way
> is another matter, so I don't think it's something worth really worrying
> about without a definite need.
I'd expect no-one would implement separate read and write IDs, given
that the IOMMU should distinguish reads and writes anyway.
Generally I don't think multiple matches for the same RID make sense.
> > +- iommu-map-mask: A mask to be applied to each Requester ID prior to being
> > + mapped to an iommu-specifier per the iommu-map property.
>
> Am I right to assume a mask of 0 would be a valid way to represent
> "everything" (and if so, should rid-base and length just be ignored, or
> mandated to be 0 and 1 respectively)? It looks a bit off at first
> glance, but it does neatly address a genuine use-case.
I think that should be valid, yes.
> > +
> > +
> > +Example (1)
> > +===========
> > +
> > +/ {
> > + #address-cells = <1>;
> > + #size-cells = <1>;
> > +
> > + iommu: iommu@a {
> > + reg = <0xa 0x1>;
> > + compatible = "vendor,some-iommu";
> > + #iommu-cells = <1>;
>
> Troll question; what do we do when #iommu-cells > 1, where the IOMMU is
> expecting some extra data associated with each ID (say, memory attributes)?
It really depends on the format of your iommu-specifier.
In the degenerate case, you can simply provide each RID with its own
translation and a full iommu-specifier, though that could take an
appeciable amount of space.
I would hope that for PCI you shouldn't need to describe the
transformation of memory attributes, however.
Thanks,
Mark.
looks good. possible to describe the chip we have.
> On Jul 23, 2015, at 9:52 AM, Mark Rutland <[email protected]> wrote:
>
> Currently msi-parent is used by a few bindings to describe the
> relationship between a PCI root complex and a single MSI controller, but
> this property does not have a generic binding document.
>
> Additionally, msi-parent is insufficient to describe more complex
> relationships between MSI controllers and devices under a root complex,
> where devices may be able to target multiple MSI controllers, or where
> MSI controllers use (non-probeable) sideband information to distinguish
> devices.
>
> This patch adds a generic binding for mapping PCI devices to MSI
> controllers. This document covers msi-parent, and a new msi-map property
> (specific to PCI*) which may be used to map devices (identified by their
> Requester ID) to sideband data for each MSI controller that they may
> target.
>
> Signed-off-by: Mark Rutland <[email protected]>
> ---
> Documentation/devicetree/bindings/pci/pci-msi.txt | 220 ++++++++++++++++++++++
> 1 file changed, 220 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/pci/pci-msi.txt
>
> diff --git a/Documentation/devicetree/bindings/pci/pci-msi.txt b/Documentation/devicetree/bindings/pci/pci-msi.txt
> new file mode 100644
> index 0000000..9b3cc81
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/pci/pci-msi.txt
> @@ -0,0 +1,220 @@
> +This document describes the generic device tree binding for describing the
> +relationship between PCI devices and MSI controllers.
> +
> +Each PCI device under a root complex is uniquely identified by its Requester ID
> +(AKA RID). A Requester ID is a triplet of a Bus number, Device number, and
> +Function number.
> +
> +For the purpose of this document, when treated as a numeric value, a RID is
> +formatted such that:
> +
> +* Bits [15:8] are the Bus number.
> +* Bits [7:3] are the Device number.
> +* Bits [2:0] are the Function number.
> +* Any other bits required for padding must be zero.
> +
> +MSIs may be distinguished in part through the use of sideband data accompanying
> +writes. In the case of PCI devices, this sideband data may be derived from the
> +Requester ID. A mechanism is required to associate a device with both the MSI
> +controllers it can address, and the sideband data that will be associated with
> +its writes to those controllers.
> +
> +For generic MSI bindings, see
> +Documentation/devicetree/bindings/interrupt-controller/msi.txt.
> +
> +
> +PCI root complex
> +================
> +
> +Optional properties
> +-------------------
> +
> +- msi-map: Maps a Requester ID to an MSI controller and associated
> + msi-specifier data. The property is an arbitrary number of tuples of
> + (rid-base,msi-controller,msi-base,length), where:
> +
> + * rid-base is a single cell describing the first RID matched by the entry.
> +
> + * msi-controller is a single phandle to an MSI controller
> +
> + * msi-base is an msi-specifier describing the msi-specifier produced for the
> + first RID matched by the entry.
> +
> + * length is a single cell describing how many consecutive RIDs are matched
> + following the rid-base.
> +
> + Any RID r in the interval [rid-base, rid-base + length) is associated with
> + the listed msi-controller, with the msi-specifier (r - rid-base + msi-base).
> +
> +- msi-map-mask: A mask to be applied to each Requester ID prior to being mapped
> + to an msi-specifier per the msi-map property.
> +
> +- msi-parent: Describes the MSI parent of the root complex itself. Where
> + the root complex and MSI controller do not pass sideband data with MSI
> + writes, this property may be used to describe the MSI controller(s)
> + used by PCI devices under the root complex, if defined as such in the
> + binding for the root complex.
> +
> +
> +Example (1)
> +===========
> +
> +/ {
> + #address-cells = <1>;
> + #size-cells = <1>;
> +
> + msi: msi-controller@a {
> + reg = <0xa 0x1>;
> + compatible = "vendor,some-controller";
> + msi-controller;
> + #msi-cells = <1>;
> + };
> +
> + pci: pci@f {
> + reg = <0xf 0x1>;
> + compatible = "vendor,pcie-root-complex";
> + device_type = "pci";
> +
> + /*
> + * The sideband data provided to the MSI controller is
> + * the RID, identity-mapped.
> + */
> + msi-map = <0x0 &msi_a 0x0 0x10000>,
> + };
> +};
> +
> +
> +Example (2)
> +===========
> +
> +/ {
> + #address-cells = <1>;
> + #size-cells = <1>;
> +
> + msi: msi-controller@a {
> + reg = <0xa 0x1>;
> + compatible = "vendor,some-controller";
> + msi-controller;
> + #msi-cells = <1>;
> + };
> +
> + pci: pci@f {
> + reg = <0xf 0x1>;
> + compatible = "vendor,pcie-root-complex";
> + device_type = "pci";
> +
> + /*
> + * The sideband data provided to the MSI controller is
> + * the RID, masked to only the device and function bits.
> + */
> + msi-map = <0x0 &msi_a 0x0 0x100>,
> + msi-map-mask = <0xff>
> + };
> +};
> +
> +
> +Example (3)
> +===========
> +
> +/ {
> + #address-cells = <1>;
> + #size-cells = <1>;
> +
> + msi: msi-controller@a {
> + reg = <0xa 0x1>;
> + compatible = "vendor,some-controller";
> + msi-controller;
> + #msi-cells = <1>;
> + };
> +
> + pci: pci@f {
> + reg = <0xf 0x1>;
> + compatible = "vendor,pcie-root-complex";
> + device_type = "pci";
> +
> + /*
> + * The sideband data provided to the MSI controller is
> + * the RID, but the high bit of the bus number is
> + * ignored.
> + */
> + msi-map = <0x0000 &msi 0x0000 0x8000>,
> + <0x8000 &msi 0x0000 0x8000>;
> + };
> +};
> +
> +
> +Example (4)
> +===========
> +
> +/ {
> + #address-cells = <1>;
> + #size-cells = <1>;
> +
> + msi: msi-controller@a {
> + reg = <0xa 0x1>;
> + compatible = "vendor,some-controller";
> + msi-controller;
> + #msi-cells = <1>;
> + };
> +
> + pci: pci@f {
> + reg = <0xf 0x1>;
> + compatible = "vendor,pcie-root-complex";
> + device_type = "pci";
> +
> + /*
> + * The sideband data provided to the MSI controller is
> + * the RID, but the high bit of the bus number is
> + * negated.
> + */
> + msi-map = <0x0000 &msi 0x8000 0x8000>,
> + <0x8000 &msi 0x0000 0x8000>;
> + };
> +};
> +
> +
> +Example (5)
> +===========
> +
> +/ {
> + #address-cells = <1>;
> + #size-cells = <1>;
> +
> + msi_a: msi-controller@a {
> + reg = <0xa 0x1>;
> + compatible = "vendor,some-controller";
> + msi-controller;
> + #msi-cells = <1>;
> + };
> +
> + msi_b: msi-controller@b {
> + reg = <0xb 0x1>;
> + compatible = "vendor,some-controller";
> + msi-controller;
> + #msi-cells = <1>;
> + };
> +
> + msi_c: msi-controller@c {
> + reg = <0xc 0x1>;
> + compatible = "vendor,some-controller";
> + msi-controller;
> + #msi-cells = <1>;
> + };
> +
> + pci: pci@c {
> + reg = <0xf 0x1>;
> + compatible = "vendor,pcie-root-complex";
> + device_type = "pci";
> +
> + /*
> + * The sideband data provided to MSI controller a is the
> + * RID, but the high bit of the bus number is negated.
> + * The sideband data provided to MSI controller b is the
> + * RID, identity-mapped.
> + * MSI controller c is not addressable.
> + */
> + msi-map = <0x0000 &msi_a 0x8000 0x08000>,
> + <0x8000 &msi_a 0x0000 0x08000>,
> + <0x0000 &msi_b 0x0000 0x10000>;
> + };
they can be identical right? like
<0x8000 &msi_a 0x0000 0x08000>,
<0x8000 &msi_b 0x0000 0x08000>;
Thanks,
Tirumalesh.
> +};
> --
> 1.9.1
>
Hi Mark,
On 23/07/15 17:52, Mark Rutland wrote:
> Currently msi-parent is used in a couple of drivers despite being fairly
> underspecified. This patch adds a generic binding for MSIs (including
> the existing msi-parent property) enabling the description of platform
> devices capable of using MSIs.
>
> While MSIs are primarily distinguished by doorbell and payload, some MSI
> controllers (e.g. the GICv3 ITS) also use side-band information
> accompanying the write to identify the master which originated the MSI,
> to allow for sandboxing. This sideband information is non-probeable and
> needs to be described in the DT. Other MSI controllers may have
> additional configuration details which need to be described per-master.
>
> This patch adds a generic msi-parent binding document, extending the
> de-facto standard with a new (optional) #msi-cells which can be used to
> express any per-master configuration and/or sideband data. This is
> sufficient to describe non-hotpluggable devices.
>
> For busses where sideband data may be derived from some bus-specific
> master ID scheme, other properties will be required to describe the
> mapping.
>
> Signed-off-by: Mark Rutland <[email protected]>
> ---
> .../bindings/interrupt-controller/msi.txt | 135 +++++++++++++++++++++
> 1 file changed, 135 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/interrupt-controller/msi.txt
>
> diff --git a/Documentation/devicetree/bindings/interrupt-controller/msi.txt b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
> new file mode 100644
> index 0000000..c60c034
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
> @@ -0,0 +1,135 @@
> +This document describes the generic device tree binding for MSI controllers and
> +their master(s).
> +
> +Message Signaled Interrupts (MSIs) are a class of interrupts generated by a
> +write to an MMIO address.
> +
> +MSIs were originally specified by PCI (and are used with PCIe), but may also be
> +used with other busses, and hence a mechanism is required to relate devices on
> +those busses to the MSI controllers which they are capable of using,
> +potentially including additional information.
> +
> +MSIs are distinguished by some combination of:
> +
> +- The doorbell (the MMIO address written to).
> +
> + Devices may be configured by software to write to arbitrary doorbells which
> + they can address. An MSI controller may feature a number of doorbells.
> +
> +- The payload (the value written to the doorbell).
> +
> + Devices may be configured to write an arbitrary payload chosen by software.
> + MSI controllers may have restrictions on permitted payloads.
> +
> +- Sideband information accompanying the write.
> +
> + Typically this is neither configurable nor probeable, and depends on the path
> + taken through the memory system (i.e. it is a property of the combination of
> + MSI controller and device rather than a property of either in isolation).
> +
> +
> +MSI controllers:
> +================
> +
> +An MSI controller signals interrupts to a CPU when a write is made to an MMIO
> +address by some master. An MSI controller may feature a number of doorbells.
> +
> +Required properties:
> +--------------------
> +
> +- msi-controller: Identifies the node as an MSI controller.
> +
> +Optional properties:
> +--------------------
> +
> +- #msi-cells: The number of cells in an msi-specifier, required if not zero.
> +
> + Typically this will encode information related to sideband data, and will
> + not encode doorbells or payloads as these can be configured dynamically.
> +
> + The meaning of the msi-specifier is defined by the device tree binding of
> + the specific MSI controller.
> +
> +
> +MSI clients
> +===========
> +
> +MSI clients are devices which generate MSIs. For each MSI they wish to
> +generate, the doorbell and payload may be configured, though sideband
> +information may not be configurable.
> +
> +Required properties:
> +--------------------
> +
> +- msi-parent: A list of phandle + msi-specifier pairs, one for each MSI
> + controller which the device is capable of using.
> +
> + This property is unordered, and MSIs may be allocated from any combination of
> + MSI controllers listed in the msi-parent property.
> +
> + If a device has restrictions on the allocation of MSIs, these restrictions
> + must be described with additional properties.
> +
> + When #msi-cells is non-zero, busses with an msi-parent will require
> + additional properties to describe the relationship between devices on the bus
> + and the set of MSIs they can potentially generate.
> +
> +
> +Example
> +=======
> +
> +/ {
> + #address-cells = <1>;
> + #size-cells = <1>;
> +
> + msi_a: msi-controller@a {
> + reg = <0xa 0xf00>;
> + compatible = "vendor-a,some-controller";
> + msi-controller;
> + /* No sideband data, so #msi-cells omitted */
> + };
> +
> + msi_b: msi-controller@b {
> + reg = <0xb 0xf00>;
> + compatible = "vendor-b,another-controller";
> + msi-controller;
> + /* Each device has some unique ID */
> + #msi-cells = <1>;
> + };
> +
> + msi_c: msi-controller@c {
> + reg = <0xb 0xf00>;
> + compatible = "vendor-b,another-controller";
> + msi-controller;
> + /* Each device has some unique ID */
> + #msi-cells = <1>;
> + };
> +
> + dev@0 {
> + reg = <0x0 0xf00>;
> + compatible = "vendor-c,some-device";
> +
> + /* Can only generate MSIs to msi_a */
> + msi-parent = <&msi_a>;
> + };
> +
> + dev@1 {
> + reg = <0x1 0xf00>;
> + compatible = "vendor-c,some-device";
> +
> + /*
> + * Can generate MSIs to either A or B.
> + */
> + msi-parent = <&msi_a>, <&msi_b 0x17>;
> + };
> +
> + dev@2 {
> + reg = <0x2 0xf00>;
> + compatible = "vendor-c,some-device";
> + /*
> + * Has different IDs at each MSI controller.
> + * Can generate MSIs to all of the MSI controllers.
> + */
> + msi-parent = <&msi_a>, <&msi_b 0x17>, <&msi_c 0x53>;
> + };
> +};
>
This looks quite good for the non-PCI stuff. Should you also cover the
PCI usage of msi-parent? I'm can't really see the meaning of #msi-cells
in that context. Should it be entirely ignored? OR did you have some
specific usage in mind?
Thanks,
M.
--
Jazz is not dead. It just smells funny...
On 23/07/15 17:52, Mark Rutland wrote:
> Currently msi-parent is used by a few bindings to describe the
> relationship between a PCI root complex and a single MSI controller, but
> this property does not have a generic binding document.
>
> Additionally, msi-parent is insufficient to describe more complex
> relationships between MSI controllers and devices under a root complex,
> where devices may be able to target multiple MSI controllers, or where
> MSI controllers use (non-probeable) sideband information to distinguish
> devices.
>
> This patch adds a generic binding for mapping PCI devices to MSI
> controllers. This document covers msi-parent, and a new msi-map property
> (specific to PCI*) which may be used to map devices (identified by their
> Requester ID) to sideband data for each MSI controller that they may
> target.
>
> Signed-off-by: Mark Rutland <[email protected]>
> ---
> Documentation/devicetree/bindings/pci/pci-msi.txt | 220 ++++++++++++++++++++++
> 1 file changed, 220 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/pci/pci-msi.txt
>
> diff --git a/Documentation/devicetree/bindings/pci/pci-msi.txt b/Documentation/devicetree/bindings/pci/pci-msi.txt
> new file mode 100644
> index 0000000..9b3cc81
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/pci/pci-msi.txt
> @@ -0,0 +1,220 @@
> +This document describes the generic device tree binding for describing the
> +relationship between PCI devices and MSI controllers.
> +
> +Each PCI device under a root complex is uniquely identified by its Requester ID
> +(AKA RID). A Requester ID is a triplet of a Bus number, Device number, and
> +Function number.
> +
> +For the purpose of this document, when treated as a numeric value, a RID is
> +formatted such that:
> +
> +* Bits [15:8] are the Bus number.
> +* Bits [7:3] are the Device number.
> +* Bits [2:0] are the Function number.
> +* Any other bits required for padding must be zero.
> +
> +MSIs may be distinguished in part through the use of sideband data accompanying
> +writes. In the case of PCI devices, this sideband data may be derived from the
> +Requester ID. A mechanism is required to associate a device with both the MSI
> +controllers it can address, and the sideband data that will be associated with
> +its writes to those controllers.
> +
> +For generic MSI bindings, see
> +Documentation/devicetree/bindings/interrupt-controller/msi.txt.
> +
> +
> +PCI root complex
> +================
> +
> +Optional properties
> +-------------------
> +
> +- msi-map: Maps a Requester ID to an MSI controller and associated
> + msi-specifier data. The property is an arbitrary number of tuples of
> + (rid-base,msi-controller,msi-base,length), where:
> +
> + * rid-base is a single cell describing the first RID matched by the entry.
> +
> + * msi-controller is a single phandle to an MSI controller
> +
> + * msi-base is an msi-specifier describing the msi-specifier produced for the
> + first RID matched by the entry.
> +
> + * length is a single cell describing how many consecutive RIDs are matched
> + following the rid-base.
> +
> + Any RID r in the interval [rid-base, rid-base + length) is associated with
> + the listed msi-controller, with the msi-specifier (r - rid-base + msi-base).
> +
> +- msi-map-mask: A mask to be applied to each Requester ID prior to being mapped
> + to an msi-specifier per the msi-map property.
> +
> +- msi-parent: Describes the MSI parent of the root complex itself. Where
> + the root complex and MSI controller do not pass sideband data with MSI
> + writes, this property may be used to describe the MSI controller(s)
> + used by PCI devices under the root complex, if defined as such in the
> + binding for the root complex.
Right, this is where I'd expect some details about #msi-cells. Is it
meant to be ignored? The lack of symmetry between the PCI and non-PCI
use cases feels a bit inelegant (not to mention that it precludes having
an unified parser for both cases).
This otherwise looks good to me.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
> > +Example (5)
> > +===========
> > +
> > +/ {
> > + #address-cells = <1>;
> > + #size-cells = <1>;
> > +
> > + msi_a: msi-controller@a {
> > + reg = <0xa 0x1>;
> > + compatible = "vendor,some-controller";
> > + msi-controller;
> > + #msi-cells = <1>;
> > + };
> > +
> > + msi_b: msi-controller@b {
> > + reg = <0xb 0x1>;
> > + compatible = "vendor,some-controller";
> > + msi-controller;
> > + #msi-cells = <1>;
> > + };
> > +
> > + msi_c: msi-controller@c {
> > + reg = <0xc 0x1>;
> > + compatible = "vendor,some-controller";
> > + msi-controller;
> > + #msi-cells = <1>;
> > + };
> > +
> > + pci: pci@c {
> > + reg = <0xf 0x1>;
> > + compatible = "vendor,pcie-root-complex";
> > + device_type = "pci";
> > +
> > + /*
> > + * The sideband data provided to MSI controller a is the
> > + * RID, but the high bit of the bus number is negated.
> > + * The sideband data provided to MSI controller b is the
> > + * RID, identity-mapped.
> > + * MSI controller c is not addressable.
> > + */
> > + msi-map = <0x0000 &msi_a 0x8000 0x08000>,
> > + <0x8000 &msi_a 0x0000 0x08000>,
> > + <0x0000 &msi_b 0x0000 0x10000>;
> > + };
>
> they can be identical right? like
> <0x8000 &msi_a 0x0000 0x08000>,
> <0x8000 &msi_b 0x0000 0x08000>;
In general that would be valid, yes.
In this case two entries are required for MSI controller a because the
high bit passed to it is negated. This does not occur for MSI controller
b, so it only requires a single entry to describe the transformation.
Thanks,
Mark.
On Mon, Jul 27, 2015 at 09:02:46AM +0100, Marc Zyngier wrote:
> Hi Mark,
Hi,
> On 23/07/15 17:52, Mark Rutland wrote:
> > Currently msi-parent is used in a couple of drivers despite being fairly
> > underspecified. This patch adds a generic binding for MSIs (including
> > the existing msi-parent property) enabling the description of platform
> > devices capable of using MSIs.
> >
> > While MSIs are primarily distinguished by doorbell and payload, some MSI
> > controllers (e.g. the GICv3 ITS) also use side-band information
> > accompanying the write to identify the master which originated the MSI,
> > to allow for sandboxing. This sideband information is non-probeable and
> > needs to be described in the DT. Other MSI controllers may have
> > additional configuration details which need to be described per-master.
> >
> > This patch adds a generic msi-parent binding document, extending the
> > de-facto standard with a new (optional) #msi-cells which can be used to
> > express any per-master configuration and/or sideband data. This is
> > sufficient to describe non-hotpluggable devices.
> >
> > For busses where sideband data may be derived from some bus-specific
> > master ID scheme, other properties will be required to describe the
> > mapping.
> >
> > Signed-off-by: Mark Rutland <[email protected]>
> > ---
> > .../bindings/interrupt-controller/msi.txt | 135 +++++++++++++++++++++
> > 1 file changed, 135 insertions(+)
> > create mode 100644 Documentation/devicetree/bindings/interrupt-controller/msi.txt
> >
> > diff --git a/Documentation/devicetree/bindings/interrupt-controller/msi.txt b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
> > new file mode 100644
> > index 0000000..c60c034
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
> > @@ -0,0 +1,135 @@
> > +This document describes the generic device tree binding for MSI controllers and
> > +their master(s).
> > +
> > +Message Signaled Interrupts (MSIs) are a class of interrupts generated by a
> > +write to an MMIO address.
> > +
> > +MSIs were originally specified by PCI (and are used with PCIe), but may also be
> > +used with other busses, and hence a mechanism is required to relate devices on
> > +those busses to the MSI controllers which they are capable of using,
> > +potentially including additional information.
> > +
> > +MSIs are distinguished by some combination of:
> > +
> > +- The doorbell (the MMIO address written to).
> > +
> > + Devices may be configured by software to write to arbitrary doorbells which
> > + they can address. An MSI controller may feature a number of doorbells.
> > +
> > +- The payload (the value written to the doorbell).
> > +
> > + Devices may be configured to write an arbitrary payload chosen by software.
> > + MSI controllers may have restrictions on permitted payloads.
> > +
> > +- Sideband information accompanying the write.
> > +
> > + Typically this is neither configurable nor probeable, and depends on the path
> > + taken through the memory system (i.e. it is a property of the combination of
> > + MSI controller and device rather than a property of either in isolation).
> > +
> > +
> > +MSI controllers:
> > +================
> > +
> > +An MSI controller signals interrupts to a CPU when a write is made to an MMIO
> > +address by some master. An MSI controller may feature a number of doorbells.
> > +
> > +Required properties:
> > +--------------------
> > +
> > +- msi-controller: Identifies the node as an MSI controller.
> > +
> > +Optional properties:
> > +--------------------
> > +
> > +- #msi-cells: The number of cells in an msi-specifier, required if not zero.
> > +
> > + Typically this will encode information related to sideband data, and will
> > + not encode doorbells or payloads as these can be configured dynamically.
> > +
> > + The meaning of the msi-specifier is defined by the device tree binding of
> > + the specific MSI controller.
> > +
> > +
> > +MSI clients
> > +===========
> > +
> > +MSI clients are devices which generate MSIs. For each MSI they wish to
> > +generate, the doorbell and payload may be configured, though sideband
> > +information may not be configurable.
> > +
> > +Required properties:
> > +--------------------
> > +
> > +- msi-parent: A list of phandle + msi-specifier pairs, one for each MSI
> > + controller which the device is capable of using.
> > +
> > + This property is unordered, and MSIs may be allocated from any combination of
> > + MSI controllers listed in the msi-parent property.
> > +
> > + If a device has restrictions on the allocation of MSIs, these restrictions
> > + must be described with additional properties.
> > +
> > + When #msi-cells is non-zero, busses with an msi-parent will require
> > + additional properties to describe the relationship between devices on the bus
> > + and the set of MSIs they can potentially generate.
> > +
> > +
> > +Example
> > +=======
> > +
> > +/ {
> > + #address-cells = <1>;
> > + #size-cells = <1>;
> > +
> > + msi_a: msi-controller@a {
> > + reg = <0xa 0xf00>;
> > + compatible = "vendor-a,some-controller";
> > + msi-controller;
> > + /* No sideband data, so #msi-cells omitted */
> > + };
> > +
> > + msi_b: msi-controller@b {
> > + reg = <0xb 0xf00>;
> > + compatible = "vendor-b,another-controller";
> > + msi-controller;
> > + /* Each device has some unique ID */
> > + #msi-cells = <1>;
> > + };
> > +
> > + msi_c: msi-controller@c {
> > + reg = <0xb 0xf00>;
> > + compatible = "vendor-b,another-controller";
> > + msi-controller;
> > + /* Each device has some unique ID */
> > + #msi-cells = <1>;
> > + };
> > +
> > + dev@0 {
> > + reg = <0x0 0xf00>;
> > + compatible = "vendor-c,some-device";
> > +
> > + /* Can only generate MSIs to msi_a */
> > + msi-parent = <&msi_a>;
> > + };
> > +
> > + dev@1 {
> > + reg = <0x1 0xf00>;
> > + compatible = "vendor-c,some-device";
> > +
> > + /*
> > + * Can generate MSIs to either A or B.
> > + */
> > + msi-parent = <&msi_a>, <&msi_b 0x17>;
> > + };
> > +
> > + dev@2 {
> > + reg = <0x2 0xf00>;
> > + compatible = "vendor-c,some-device";
> > + /*
> > + * Has different IDs at each MSI controller.
> > + * Can generate MSIs to all of the MSI controllers.
> > + */
> > + msi-parent = <&msi_a>, <&msi_b 0x17>, <&msi_c 0x53>;
> > + };
> > +};
> >
>
> This looks quite good for the non-PCI stuff. Should you also cover the
> PCI usage of msi-parent?
As far as I can tell, the current PCI usage of msi-parent is practically
identical to the generic usage, with the proviso that the devices under
a PCI root complex are assumed to be indistinguishable from the root
complex from the PoV of the MSI controller.
So I'm not sure what would be relevant to describe here.
> I'm can't really see the meaning of #msi-cells in that context. Should
> it be entirely ignored? OR did you have some specific usage in mind?
It shouldn't be ignored; if the MSI controller has a non-zero #msi-cells
then it requires that information to operate correctly (e.g. to
distinguish masters).
It may simply be that all devices under the root complex are
indistinguishable from each other, but can be distinguished form other
devices in the system using the same MSI controller.
I also imagine that there may be PCI root complexes which signal their
own management interrupts as MSIs. Such a root complex needs msi-parent
to describe its relationship with the MSI controller, which is distinct
from the relationship between its children and the MSI controller.
Thanks,
Mark.
On 27/07/15 10:46, Mark Rutland wrote:
> On Mon, Jul 27, 2015 at 09:02:46AM +0100, Marc Zyngier wrote:
>> Hi Mark,
>
> Hi,
>
>> On 23/07/15 17:52, Mark Rutland wrote:
>>> Currently msi-parent is used in a couple of drivers despite being fairly
>>> underspecified. This patch adds a generic binding for MSIs (including
>>> the existing msi-parent property) enabling the description of platform
>>> devices capable of using MSIs.
>>>
>>> While MSIs are primarily distinguished by doorbell and payload, some MSI
>>> controllers (e.g. the GICv3 ITS) also use side-band information
>>> accompanying the write to identify the master which originated the MSI,
>>> to allow for sandboxing. This sideband information is non-probeable and
>>> needs to be described in the DT. Other MSI controllers may have
>>> additional configuration details which need to be described per-master.
>>>
>>> This patch adds a generic msi-parent binding document, extending the
>>> de-facto standard with a new (optional) #msi-cells which can be used to
>>> express any per-master configuration and/or sideband data. This is
>>> sufficient to describe non-hotpluggable devices.
>>>
>>> For busses where sideband data may be derived from some bus-specific
>>> master ID scheme, other properties will be required to describe the
>>> mapping.
>>>
>>> Signed-off-by: Mark Rutland <[email protected]>
>>> ---
>>> .../bindings/interrupt-controller/msi.txt | 135 +++++++++++++++++++++
>>> 1 file changed, 135 insertions(+)
>>> create mode 100644 Documentation/devicetree/bindings/interrupt-controller/msi.txt
>>>
>>> diff --git a/Documentation/devicetree/bindings/interrupt-controller/msi.txt b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
>>> new file mode 100644
>>> index 0000000..c60c034
>>> --- /dev/null
>>> +++ b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
>>> @@ -0,0 +1,135 @@
>>> +This document describes the generic device tree binding for MSI controllers and
>>> +their master(s).
>>> +
>>> +Message Signaled Interrupts (MSIs) are a class of interrupts generated by a
>>> +write to an MMIO address.
>>> +
>>> +MSIs were originally specified by PCI (and are used with PCIe), but may also be
>>> +used with other busses, and hence a mechanism is required to relate devices on
>>> +those busses to the MSI controllers which they are capable of using,
>>> +potentially including additional information.
>>> +
>>> +MSIs are distinguished by some combination of:
>>> +
>>> +- The doorbell (the MMIO address written to).
>>> +
>>> + Devices may be configured by software to write to arbitrary doorbells which
>>> + they can address. An MSI controller may feature a number of doorbells.
>>> +
>>> +- The payload (the value written to the doorbell).
>>> +
>>> + Devices may be configured to write an arbitrary payload chosen by software.
>>> + MSI controllers may have restrictions on permitted payloads.
>>> +
>>> +- Sideband information accompanying the write.
>>> +
>>> + Typically this is neither configurable nor probeable, and depends on the path
>>> + taken through the memory system (i.e. it is a property of the combination of
>>> + MSI controller and device rather than a property of either in isolation).
>>> +
>>> +
>>> +MSI controllers:
>>> +================
>>> +
>>> +An MSI controller signals interrupts to a CPU when a write is made to an MMIO
>>> +address by some master. An MSI controller may feature a number of doorbells.
>>> +
>>> +Required properties:
>>> +--------------------
>>> +
>>> +- msi-controller: Identifies the node as an MSI controller.
>>> +
>>> +Optional properties:
>>> +--------------------
>>> +
>>> +- #msi-cells: The number of cells in an msi-specifier, required if not zero.
>>> +
>>> + Typically this will encode information related to sideband data, and will
>>> + not encode doorbells or payloads as these can be configured dynamically.
>>> +
>>> + The meaning of the msi-specifier is defined by the device tree binding of
>>> + the specific MSI controller.
>>> +
>>> +
>>> +MSI clients
>>> +===========
>>> +
>>> +MSI clients are devices which generate MSIs. For each MSI they wish to
>>> +generate, the doorbell and payload may be configured, though sideband
>>> +information may not be configurable.
>>> +
>>> +Required properties:
>>> +--------------------
>>> +
>>> +- msi-parent: A list of phandle + msi-specifier pairs, one for each MSI
>>> + controller which the device is capable of using.
>>> +
>>> + This property is unordered, and MSIs may be allocated from any combination of
>>> + MSI controllers listed in the msi-parent property.
>>> +
>>> + If a device has restrictions on the allocation of MSIs, these restrictions
>>> + must be described with additional properties.
>>> +
>>> + When #msi-cells is non-zero, busses with an msi-parent will require
>>> + additional properties to describe the relationship between devices on the bus
>>> + and the set of MSIs they can potentially generate.
>>> +
>>> +
>>> +Example
>>> +=======
>>> +
>>> +/ {
>>> + #address-cells = <1>;
>>> + #size-cells = <1>;
>>> +
>>> + msi_a: msi-controller@a {
>>> + reg = <0xa 0xf00>;
>>> + compatible = "vendor-a,some-controller";
>>> + msi-controller;
>>> + /* No sideband data, so #msi-cells omitted */
>>> + };
>>> +
>>> + msi_b: msi-controller@b {
>>> + reg = <0xb 0xf00>;
>>> + compatible = "vendor-b,another-controller";
>>> + msi-controller;
>>> + /* Each device has some unique ID */
>>> + #msi-cells = <1>;
>>> + };
>>> +
>>> + msi_c: msi-controller@c {
>>> + reg = <0xb 0xf00>;
>>> + compatible = "vendor-b,another-controller";
>>> + msi-controller;
>>> + /* Each device has some unique ID */
>>> + #msi-cells = <1>;
>>> + };
>>> +
>>> + dev@0 {
>>> + reg = <0x0 0xf00>;
>>> + compatible = "vendor-c,some-device";
>>> +
>>> + /* Can only generate MSIs to msi_a */
>>> + msi-parent = <&msi_a>;
>>> + };
>>> +
>>> + dev@1 {
>>> + reg = <0x1 0xf00>;
>>> + compatible = "vendor-c,some-device";
>>> +
>>> + /*
>>> + * Can generate MSIs to either A or B.
>>> + */
>>> + msi-parent = <&msi_a>, <&msi_b 0x17>;
>>> + };
>>> +
>>> + dev@2 {
>>> + reg = <0x2 0xf00>;
>>> + compatible = "vendor-c,some-device";
>>> + /*
>>> + * Has different IDs at each MSI controller.
>>> + * Can generate MSIs to all of the MSI controllers.
>>> + */
>>> + msi-parent = <&msi_a>, <&msi_b 0x17>, <&msi_c 0x53>;
>>> + };
>>> +};
>>>
>>
>> This looks quite good for the non-PCI stuff. Should you also cover the
>> PCI usage of msi-parent?
>
> As far as I can tell, the current PCI usage of msi-parent is practically
> identical to the generic usage, with the proviso that the devices under
> a PCI root complex are assumed to be indistinguishable from the root
> complex from the PoV of the MSI controller.
>
> So I'm not sure what would be relevant to describe here.
>
>> I'm can't really see the meaning of #msi-cells in that context. Should
>> it be entirely ignored? OR did you have some specific usage in mind?
>
> It shouldn't be ignored; if the MSI controller has a non-zero #msi-cells
> then it requires that information to operate correctly (e.g. to
> distinguish masters).
>
> It may simply be that all devices under the root complex are
> indistinguishable from each other, but can be distinguished form other
> devices in the system using the same MSI controller.
>
> I also imagine that there may be PCI root complexes which signal their
> own management interrupts as MSIs. Such a root complex needs msi-parent
> to describe its relationship with the MSI controller, which is distinct
> from the relationship between its children and the MSI controller.
Looks tortuous, but why not... ;-)
As an aside, the GICv3 ITS part of the non-PCI MSI support which is
queued for 4.3 is using now this binding to find out about the deviceID,
and I believe Ma Jun is going to use this for his mbigen driver.
I also have additional patches for the core code to parse some of the
properties.
Thanks,
M.
It would be good if we could make some forward progress to merge this
--
Jazz is not dead. It just smells funny...
Hi Mark
Thanks for the patch. Please find my comment inline.
Regards
Varun
> -----Original Message-----
> From: [email protected] [mailto:iommu-
> [email protected]] On Behalf Of Mark Rutland
> Sent: Thursday, July 23, 2015 10:23 PM
> To: [email protected]
> Cc: Mark Rutland; [email protected]; [email protected];
> [email protected]; [email protected]; linux-
> [email protected]; [email protected]; [email protected]
> foundation.org; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]
> Subject: [PATCH 2/3] Docs: dt: Add PCI MSI map bindings
>
> Currently msi-parent is used by a few bindings to describe the relationship
> between a PCI root complex and a single MSI controller, but this property
> does not have a generic binding document.
>
> Additionally, msi-parent is insufficient to describe more complex
> relationships between MSI controllers and devices under a root complex,
> where devices may be able to target multiple MSI controllers, or where MSI
> controllers use (non-probeable) sideband information to distinguish devices.
>
> This patch adds a generic binding for mapping PCI devices to MSI controllers.
> This document covers msi-parent, and a new msi-map property (specific to
> PCI*) which may be used to map devices (identified by their Requester ID) to
> sideband data for each MSI controller that they may target.
>
> Signed-off-by: Mark Rutland <[email protected]>
> ---
> Documentation/devicetree/bindings/pci/pci-msi.txt | 220
> ++++++++++++++++++++++
> 1 file changed, 220 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/pci/pci-msi.txt
>
> diff --git a/Documentation/devicetree/bindings/pci/pci-msi.txt
> b/Documentation/devicetree/bindings/pci/pci-msi.txt
> new file mode 100644
> index 0000000..9b3cc81
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/pci/pci-msi.txt
> @@ -0,0 +1,220 @@
> +This document describes the generic device tree binding for describing
> +the relationship between PCI devices and MSI controllers.
> +
> +Each PCI device under a root complex is uniquely identified by its
> +Requester ID (AKA RID). A Requester ID is a triplet of a Bus number,
> +Device number, and Function number.
> +
> +For the purpose of this document, when treated as a numeric value, a
> +RID is formatted such that:
> +
> +* Bits [15:8] are the Bus number.
> +* Bits [7:3] are the Device number.
> +* Bits [2:0] are the Function number.
> +* Any other bits required for padding must be zero.
> +
> +MSIs may be distinguished in part through the use of sideband data
> +accompanying writes. In the case of PCI devices, this sideband data may
> +be derived from the Requester ID. A mechanism is required to associate
> +a device with both the MSI controllers it can address, and the sideband
> +data that will be associated with its writes to those controllers.
> +
> +For generic MSI bindings, see
> +Documentation/devicetree/bindings/interrupt-controller/msi.txt.
> +
> +
> +PCI root complex
> +================
> +
> +Optional properties
> +-------------------
> +
> +- msi-map: Maps a Requester ID to an MSI controller and associated
> + msi-specifier data. The property is an arbitrary number of tuples of
> + (rid-base,msi-controller,msi-base,length), where:
[varun] How would we account for hot plug PCI devices and SR-IOV use cases, with the rid base and length? How do we take in to account for a PCIe bridge, while setting up the requestor ID base and length?
> +
> + * rid-base is a single cell describing the first RID matched by the entry.
> +
> + * msi-controller is a single phandle to an MSI controller
> +
> + * msi-base is an msi-specifier describing the msi-specifier produced for the
> + first RID matched by the entry.
> +
> + * length is a single cell describing how many consecutive RIDs are matched
> + following the rid-base.
> +
> + Any RID r in the interval [rid-base, rid-base + length) is associated
> + with the listed msi-controller, with the msi-specifier (r - rid-base + msi-
> base).
> +
> +- msi-map-mask: A mask to be applied to each Requester ID prior to
> +being mapped
> + to an msi-specifier per the msi-map property.
> +
[varun] Can you please elaborate on a use case, where this would help.
> +- msi-parent: Describes the MSI parent of the root complex itself.
> +Where
> + the root complex and MSI controller do not pass sideband data with
> +MSI
> + writes, this property may be used to describe the MSI controller(s)
> + used by PCI devices under the root complex, if defined as such in the
> + binding for the root complex.
> +
> +
> +Example (1)
> +===========
> +
> +/ {
> + #address-cells = <1>;
> + #size-cells = <1>;
> +
> + msi: msi-controller@a {
> + reg = <0xa 0x1>;
> + compatible = "vendor,some-controller";
> + msi-controller;
> + #msi-cells = <1>;
> + };
> +
> + pci: pci@f {
> + reg = <0xf 0x1>;
> + compatible = "vendor,pcie-root-complex";
> + device_type = "pci";
> +
> + /*
> + * The sideband data provided to the MSI controller is
> + * the RID, identity-mapped.
> + */
> + msi-map = <0x0 &msi_a 0x0 0x10000>,
> + };
> +};
> +
> +
> +Example (2)
> +===========
> +
> +/ {
> + #address-cells = <1>;
> + #size-cells = <1>;
> +
> + msi: msi-controller@a {
> + reg = <0xa 0x1>;
> + compatible = "vendor,some-controller";
> + msi-controller;
> + #msi-cells = <1>;
> + };
> +
> + pci: pci@f {
> + reg = <0xf 0x1>;
> + compatible = "vendor,pcie-root-complex";
> + device_type = "pci";
> +
> + /*
> + * The sideband data provided to the MSI controller is
> + * the RID, masked to only the device and function bits.
> + */
> + msi-map = <0x0 &msi_a 0x0 0x100>,
> + msi-map-mask = <0xff>
> + };
> +};
> +
> +
> +Example (3)
> +===========
> +
> +/ {
> + #address-cells = <1>;
> + #size-cells = <1>;
> +
> + msi: msi-controller@a {
> + reg = <0xa 0x1>;
> + compatible = "vendor,some-controller";
> + msi-controller;
> + #msi-cells = <1>;
> + };
> +
> + pci: pci@f {
> + reg = <0xf 0x1>;
> + compatible = "vendor,pcie-root-complex";
> + device_type = "pci";
> +
> + /*
> + * The sideband data provided to the MSI controller is
> + * the RID, but the high bit of the bus number is
> + * ignored.
> + */
> + msi-map = <0x0000 &msi 0x0000 0x8000>,
> + <0x8000 &msi 0x0000 0x8000>;
> + };
> +};
> +
> +
> +Example (4)
> +===========
> +
> +/ {
> + #address-cells = <1>;
> + #size-cells = <1>;
> +
> + msi: msi-controller@a {
> + reg = <0xa 0x1>;
> + compatible = "vendor,some-controller";
> + msi-controller;
> + #msi-cells = <1>;
> + };
> +
> + pci: pci@f {
> + reg = <0xf 0x1>;
> + compatible = "vendor,pcie-root-complex";
> + device_type = "pci";
> +
> + /*
> + * The sideband data provided to the MSI controller is
> + * the RID, but the high bit of the bus number is
> + * negated.
> + */
> + msi-map = <0x0000 &msi 0x8000 0x8000>,
> + <0x8000 &msi 0x0000 0x8000>;
> + };
> +};
> +
> +
> +Example (5)
> +===========
> +
> +/ {
> + #address-cells = <1>;
> + #size-cells = <1>;
> +
> + msi_a: msi-controller@a {
> + reg = <0xa 0x1>;
> + compatible = "vendor,some-controller";
> + msi-controller;
> + #msi-cells = <1>;
> + };
> +
> + msi_b: msi-controller@b {
> + reg = <0xb 0x1>;
> + compatible = "vendor,some-controller";
> + msi-controller;
> + #msi-cells = <1>;
> + };
> +
> + msi_c: msi-controller@c {
> + reg = <0xc 0x1>;
> + compatible = "vendor,some-controller";
> + msi-controller;
> + #msi-cells = <1>;
> + };
> +
> + pci: pci@c {
> + reg = <0xf 0x1>;
> + compatible = "vendor,pcie-root-complex";
> + device_type = "pci";
> +
> + /*
> + * The sideband data provided to MSI controller a is the
> + * RID, but the high bit of the bus number is negated.
> + * The sideband data provided to MSI controller b is the
> + * RID, identity-mapped.
> + * MSI controller c is not addressable.
> + */
> + msi-map = <0x0000 &msi_a 0x8000 0x08000>,
> + <0x8000 &msi_a 0x0000 0x08000>,
> + <0x0000 &msi_b 0x0000 0x10000>;
> + };
> +};
> --
> 1.9.1
>
> _______________________________________________
> iommu mailing list
> [email protected]
> https://lists.linuxfoundation.org/mailman/listinfo/iommu
Rob,
Do you have any objections to this, or are you happy to take this patch?
There's a user of this binding (the GICv3 ITS) queued for v4.3 already in
the tip tree, so either we either need to be ok with this binding or we
need to rework that before v4.3.
Marc, can I take it from your use of the binding that you are happy to
provide your ack?
Thanks,
Mark.
On Thu, Jul 23, 2015 at 05:52:43PM +0100, Mark Rutland wrote:
> Currently msi-parent is used in a couple of drivers despite being fairly
> underspecified. This patch adds a generic binding for MSIs (including
> the existing msi-parent property) enabling the description of platform
> devices capable of using MSIs.
>
> While MSIs are primarily distinguished by doorbell and payload, some MSI
> controllers (e.g. the GICv3 ITS) also use side-band information
> accompanying the write to identify the master which originated the MSI,
> to allow for sandboxing. This sideband information is non-probeable and
> needs to be described in the DT. Other MSI controllers may have
> additional configuration details which need to be described per-master.
>
> This patch adds a generic msi-parent binding document, extending the
> de-facto standard with a new (optional) #msi-cells which can be used to
> express any per-master configuration and/or sideband data. This is
> sufficient to describe non-hotpluggable devices.
>
> For busses where sideband data may be derived from some bus-specific
> master ID scheme, other properties will be required to describe the
> mapping.
>
> Signed-off-by: Mark Rutland <[email protected]>
> ---
> .../bindings/interrupt-controller/msi.txt | 135 +++++++++++++++++++++
> 1 file changed, 135 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/interrupt-controller/msi.txt
>
> diff --git a/Documentation/devicetree/bindings/interrupt-controller/msi.txt b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
> new file mode 100644
> index 0000000..c60c034
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
> @@ -0,0 +1,135 @@
> +This document describes the generic device tree binding for MSI controllers and
> +their master(s).
> +
> +Message Signaled Interrupts (MSIs) are a class of interrupts generated by a
> +write to an MMIO address.
> +
> +MSIs were originally specified by PCI (and are used with PCIe), but may also be
> +used with other busses, and hence a mechanism is required to relate devices on
> +those busses to the MSI controllers which they are capable of using,
> +potentially including additional information.
> +
> +MSIs are distinguished by some combination of:
> +
> +- The doorbell (the MMIO address written to).
> +
> + Devices may be configured by software to write to arbitrary doorbells which
> + they can address. An MSI controller may feature a number of doorbells.
> +
> +- The payload (the value written to the doorbell).
> +
> + Devices may be configured to write an arbitrary payload chosen by software.
> + MSI controllers may have restrictions on permitted payloads.
> +
> +- Sideband information accompanying the write.
> +
> + Typically this is neither configurable nor probeable, and depends on the path
> + taken through the memory system (i.e. it is a property of the combination of
> + MSI controller and device rather than a property of either in isolation).
> +
> +
> +MSI controllers:
> +================
> +
> +An MSI controller signals interrupts to a CPU when a write is made to an MMIO
> +address by some master. An MSI controller may feature a number of doorbells.
> +
> +Required properties:
> +--------------------
> +
> +- msi-controller: Identifies the node as an MSI controller.
> +
> +Optional properties:
> +--------------------
> +
> +- #msi-cells: The number of cells in an msi-specifier, required if not zero.
> +
> + Typically this will encode information related to sideband data, and will
> + not encode doorbells or payloads as these can be configured dynamically.
> +
> + The meaning of the msi-specifier is defined by the device tree binding of
> + the specific MSI controller.
> +
> +
> +MSI clients
> +===========
> +
> +MSI clients are devices which generate MSIs. For each MSI they wish to
> +generate, the doorbell and payload may be configured, though sideband
> +information may not be configurable.
> +
> +Required properties:
> +--------------------
> +
> +- msi-parent: A list of phandle + msi-specifier pairs, one for each MSI
> + controller which the device is capable of using.
> +
> + This property is unordered, and MSIs may be allocated from any combination of
> + MSI controllers listed in the msi-parent property.
> +
> + If a device has restrictions on the allocation of MSIs, these restrictions
> + must be described with additional properties.
> +
> + When #msi-cells is non-zero, busses with an msi-parent will require
> + additional properties to describe the relationship between devices on the bus
> + and the set of MSIs they can potentially generate.
> +
> +
> +Example
> +=======
> +
> +/ {
> + #address-cells = <1>;
> + #size-cells = <1>;
> +
> + msi_a: msi-controller@a {
> + reg = <0xa 0xf00>;
> + compatible = "vendor-a,some-controller";
> + msi-controller;
> + /* No sideband data, so #msi-cells omitted */
> + };
> +
> + msi_b: msi-controller@b {
> + reg = <0xb 0xf00>;
> + compatible = "vendor-b,another-controller";
> + msi-controller;
> + /* Each device has some unique ID */
> + #msi-cells = <1>;
> + };
> +
> + msi_c: msi-controller@c {
> + reg = <0xb 0xf00>;
> + compatible = "vendor-b,another-controller";
> + msi-controller;
> + /* Each device has some unique ID */
> + #msi-cells = <1>;
> + };
> +
> + dev@0 {
> + reg = <0x0 0xf00>;
> + compatible = "vendor-c,some-device";
> +
> + /* Can only generate MSIs to msi_a */
> + msi-parent = <&msi_a>;
> + };
> +
> + dev@1 {
> + reg = <0x1 0xf00>;
> + compatible = "vendor-c,some-device";
> +
> + /*
> + * Can generate MSIs to either A or B.
> + */
> + msi-parent = <&msi_a>, <&msi_b 0x17>;
> + };
> +
> + dev@2 {
> + reg = <0x2 0xf00>;
> + compatible = "vendor-c,some-device";
> + /*
> + * Has different IDs at each MSI controller.
> + * Can generate MSIs to all of the MSI controllers.
> + */
> + msi-parent = <&msi_a>, <&msi_b 0x17>, <&msi_c 0x53>;
> + };
> +};
> --
> 1.9.1
>
> From: Mark Rutland <[email protected]>
> Date: Thu, Jul 23, 2015 at 11:52 AM
[cut]
> diff --git a/Documentation/devicetree/bindings/pci/pci-msi.txt
> b/Documentation/devicetree/bindings/pci/pci-msi.txt
> new file mode 100644
> index 0000000..9b3cc81
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/pci/pci-msi.txt
> @@ -0,0 +1,220 @@
> +This document describes the generic device tree binding for describing the
> +relationship between PCI devices and MSI controllers.
> +
> +Each PCI device under a root complex is uniquely identified by its Requester ID
> +(AKA RID). A Requester ID is a triplet of a Bus number, Device number, and
> +Function number.
> +
> +For the purpose of this document, when treated as a numeric value, a RID is
> +formatted such that:
> +
> +* Bits [15:8] are the Bus number.
> +* Bits [7:3] are the Device number.
> +* Bits [2:0] are the Function number.
> +* Any other bits required for padding must be zero.
> +
> +MSIs may be distinguished in part through the use of sideband data accompanying
> +writes. In the case of PCI devices, this sideband data may be derived from the
> +Requester ID. A mechanism is required to associate a device with both the MSI
> +controllers it can address, and the sideband data that will be associated with
> +its writes to those controllers.
> +
> +For generic MSI bindings, see
> +Documentation/devicetree/bindings/interrupt-controller/msi.txt.
> +
> +
> +PCI root complex
> +================
> +
> +Optional properties
> +-------------------
> +
> +- msi-map: Maps a Requester ID to an MSI controller and associated
> + msi-specifier data. The property is an arbitrary number of tuples of
> + (rid-base,msi-controller,msi-base,length), where:
> +
> + * rid-base is a single cell describing the first RID matched by the entry.
> +
> + * msi-controller is a single phandle to an MSI controller
> +
> + * msi-base is an msi-specifier describing the msi-specifier produced for the
> + first RID matched by the entry.
> +
> + * length is a single cell describing how many consecutive RIDs are matched
> + following the rid-base.
> +
> + Any RID r in the interval [rid-base, rid-base + length) is associated with
> + the listed msi-controller, with the msi-specifier (r - rid-base + msi-base).
> +
> +- msi-map-mask: A mask to be applied to each Requester ID prior to being mapped
> + to an msi-specifier per the msi-map property.
Can we extend the msi-map-mask definition to say: "A mask value of 0x0 is valid
and indicates that no RIDs are _currently_ mapped to any msi-specifier."
We have an SoC with a programmable hardware table in the PCI controller that maps
requester ID to stream ID, so the overall msi-map (and iommu-map) definition fit
into that scheme. But, we would like to be able make the RID->stream-ID mapping
decision _lazily_, in Linux, based on actual usage of PCI devices.
pcie@3600000 {
compatible = "fsl,ls2085a-pcie", "snps,dw-pcie";
device_type = "pci";
...
msi-map = <0x0 &msi_a 0x7 4>,
msi-map-mask = <0x0>
};
That specifies the there are 4 stream IDs starting at stream ID 0x7,
but the requester ID's are not mapped (because the mask is 0x0).
This tells the PCI controller driver that there are 4 msi-specifiers
(e.g. stream IDs) available and what they are.
(same definition would apply to the iommu-map-mask)
Thanks,
Stuart
????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay? ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m???? ????????I?
On 05/08/15 17:51, Mark Rutland wrote:
> Rob,
>
> Do you have any objections to this, or are you happy to take this patch?
>
> There's a user of this binding (the GICv3 ITS) queued for v4.3 already in
> the tip tree, so either we either need to be ok with this binding or we
> need to rework that before v4.3.
>
> Marc, can I take it from your use of the binding that you are happy to
> provide your ack?
Definitely.
Acked-by: Marc Zyngier <[email protected]>
M.
--
Jazz is not dead. It just smells funny...
On Wed, Aug 05, 2015 at 05:39:33PM +0100, Varun Sethi wrote:
> Hi Mark
> Thanks for the patch. Please find my comment inline.
>
> Regards
> Varun
>
> > -----Original Message-----
> > From: [email protected] [mailto:iommu-
> > [email protected]] On Behalf Of Mark Rutland
> > Sent: Thursday, July 23, 2015 10:23 PM
> > To: [email protected]
> > Cc: Mark Rutland; [email protected]; [email protected];
> > [email protected]; [email protected]; linux-
> > [email protected]; [email protected]; [email protected]
> > foundation.org; [email protected];
> > [email protected]; [email protected];
> > [email protected]; [email protected];
> > [email protected]
> > Subject: [PATCH 2/3] Docs: dt: Add PCI MSI map bindings
> >
> > Currently msi-parent is used by a few bindings to describe the relationship
> > between a PCI root complex and a single MSI controller, but this property
> > does not have a generic binding document.
> >
> > Additionally, msi-parent is insufficient to describe more complex
> > relationships between MSI controllers and devices under a root complex,
> > where devices may be able to target multiple MSI controllers, or where MSI
> > controllers use (non-probeable) sideband information to distinguish devices.
> >
> > This patch adds a generic binding for mapping PCI devices to MSI controllers.
> > This document covers msi-parent, and a new msi-map property (specific to
> > PCI*) which may be used to map devices (identified by their Requester ID) to
> > sideband data for each MSI controller that they may target.
> >
> > Signed-off-by: Mark Rutland <[email protected]>
> > ---
> > Documentation/devicetree/bindings/pci/pci-msi.txt | 220
> > ++++++++++++++++++++++
> > 1 file changed, 220 insertions(+)
> > create mode 100644 Documentation/devicetree/bindings/pci/pci-msi.txt
> >
> > diff --git a/Documentation/devicetree/bindings/pci/pci-msi.txt
> > b/Documentation/devicetree/bindings/pci/pci-msi.txt
> > new file mode 100644
> > index 0000000..9b3cc81
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/pci/pci-msi.txt
> > @@ -0,0 +1,220 @@
> > +This document describes the generic device tree binding for describing
> > +the relationship between PCI devices and MSI controllers.
> > +
> > +Each PCI device under a root complex is uniquely identified by its
> > +Requester ID (AKA RID). A Requester ID is a triplet of a Bus number,
> > +Device number, and Function number.
> > +
> > +For the purpose of this document, when treated as a numeric value, a
> > +RID is formatted such that:
> > +
> > +* Bits [15:8] are the Bus number.
> > +* Bits [7:3] are the Device number.
> > +* Bits [2:0] are the Function number.
> > +* Any other bits required for padding must be zero.
> > +
> > +MSIs may be distinguished in part through the use of sideband data
> > +accompanying writes. In the case of PCI devices, this sideband data may
> > +be derived from the Requester ID. A mechanism is required to associate
> > +a device with both the MSI controllers it can address, and the sideband
> > +data that will be associated with its writes to those controllers.
> > +
> > +For generic MSI bindings, see
> > +Documentation/devicetree/bindings/interrupt-controller/msi.txt.
> > +
> > +
> > +PCI root complex
> > +================
> > +
> > +Optional properties
> > +-------------------
> > +
> > +- msi-map: Maps a Requester ID to an MSI controller and associated
> > + msi-specifier data. The property is an arbitrary number of tuples of
> > + (rid-base,msi-controller,msi-base,length), where:
> [varun] How would we account for hot plug PCI devices and SR-IOV use cases, with the rid base and length?
For hotplug, you simply need the mapping from RID to msi-specifier to be
defined in advance in the DT, for the set of RIDs that could possibly
occur.
For SR-IOV, are you asking about ARI? I should update the description of
the RID to describe that for ARI it has the format:
* Bits [15:8] are the Bus number
* Bits [7:0] are the Identifier
Other than that, the handling would be identical to the non-ARI case.
What else am I missing?
> How do we take in to account for a PCIe bridge, while setting up the requestor ID base and length?
I'm not sure I follow the question. I don't see why this is any
different to any other requester ID.
What do you see as being the problem for this case?
> > +
> > + * rid-base is a single cell describing the first RID matched by the entry.
> > +
> > + * msi-controller is a single phandle to an MSI controller
> > +
> > + * msi-base is an msi-specifier describing the msi-specifier produced for the
> > + first RID matched by the entry.
> > +
> > + * length is a single cell describing how many consecutive RIDs are matched
> > + following the rid-base.
> > +
> > + Any RID r in the interval [rid-base, rid-base + length) is associated
> > + with the listed msi-controller, with the msi-specifier (r - rid-base + msi-
> > base).
> > +
> > +- msi-map-mask: A mask to be applied to each Requester ID prior to
> > +being mapped
> > + to an msi-specifier per the msi-map property.
> > +
> [varun] Can you please elaborate on a use case, where this would help.
It may be the case that at the MSI controller's ID space is smaller
than the RID space, and so only a subset of RID bits matter. For
example, it might be the case that only the Bus ID matters.
Using the msi-map-mask allows for a much smaller msi-map for this case,
e.g.
msi-map-mask = <0xff00>;
msi-map = <0x0000 &msi 0x00 1>,
<0x0100 &msi 0x01 1>,
<0x0200 &msi 0x01 1>,
<0x0300 &msi 0x01 1>,
...
<0xff00 &msi 0xff 1>;
Rather than:
msi-map = <0x0000 &msi 0x00 1>,
<0x0001 &msi 0x00 1>,
<0x0002 &msi 0x00 1>,
<0x0003 &msi 0x00 1>,
...
<0x00ff &msi 0x00 1>,
<0x0100 &msi 0x00 1>,
<0x0101 &msi 0x00 1>,
<0x0102 &msi 0x00 1>,
<0x0103 &msi 0x00 1>,
...
<0x01ff &msi 0x00 1>,
...
<0xff00 &msi 0xff 1>,
<0xff01 &msi 0xff 1>,
<0xff02 &msi 0xff 1>,
<0xff03 &msi 0xff 1>,
....
<0xffff &msi 0xff 1>;
Or for the case that everything maps to a single ID:
msi-map-mask = <0x0000>;
msi-map = <0x0000 &msi 0x0000 1>;
Thanks,
Mark.
[...]
> > +PCI root complex
> > +================
> > +
> > +Optional properties
> > +-------------------
> > +
> > +- msi-map: Maps a Requester ID to an MSI controller and associated
> > + msi-specifier data. The property is an arbitrary number of tuples of
> > + (rid-base,msi-controller,msi-base,length), where:
> > +
> > + * rid-base is a single cell describing the first RID matched by the entry.
> > +
> > + * msi-controller is a single phandle to an MSI controller
> > +
> > + * msi-base is an msi-specifier describing the msi-specifier produced for the
> > + first RID matched by the entry.
> > +
> > + * length is a single cell describing how many consecutive RIDs are matched
> > + following the rid-base.
> > +
> > + Any RID r in the interval [rid-base, rid-base + length) is associated with
> > + the listed msi-controller, with the msi-specifier (r - rid-base + msi-base).
> > +
> > +- msi-map-mask: A mask to be applied to each Requester ID prior to being mapped
> > + to an msi-specifier per the msi-map property.
>
> Can we extend the msi-map-mask definition to say: "A mask value of 0x0 is valid
> and indicates that no RIDs are _currently_ mapped to any msi-specifier."
That would break a valid case of the mask being all zeroes.
Consider the case that all RIDs get mapped to a single msi-specifier;
the obvious way to write that is:
msi-map-mask = <0x0000>;
msi-map = <0x0000 &msi (msi-specifier) 1>;
In this case all RIDS are always mapped to the single msi-specifier.
> We have an SoC with a programmable hardware table in the PCI controller that maps
> requester ID to stream ID, so the overall msi-map (and iommu-map) definition fit
> into that scheme. But, we would like to be able make the RID->stream-ID mapping
> decision _lazily_, in Linux, based on actual usage of PCI devices.
Dynamically programming the mapping is at odds to this binding. I don't
see how that can fit.
Why can the RID->SID mapping not be statically configured prior to
entering the OS?
Thanks,
Mark.
> -----Original Message-----
> From: Mark Rutland [mailto:[email protected]]
> Sent: Thursday, August 06, 2015 1:15 PM
> To: Yoder Stuart-B08248; Marc Zyngier; Will Deacon
> Cc: [email protected]; Lorenzo Pieralisi; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected]; linux-arm-
> [email protected]; [email protected]
> Subject: Re: [PATCH 2/3] Docs: dt: Add PCI MSI map bindings
>
> [...]
>
> > > +PCI root complex
> > > +================
> > > +
> > > +Optional properties
> > > +-------------------
> > > +
> > > +- msi-map: Maps a Requester ID to an MSI controller and associated
> > > + msi-specifier data. The property is an arbitrary number of tuples of
> > > + (rid-base,msi-controller,msi-base,length), where:
> > > +
> > > + * rid-base is a single cell describing the first RID matched by the entry.
> > > +
> > > + * msi-controller is a single phandle to an MSI controller
> > > +
> > > + * msi-base is an msi-specifier describing the msi-specifier produced for the
> > > + first RID matched by the entry.
> > > +
> > > + * length is a single cell describing how many consecutive RIDs are matched
> > > + following the rid-base.
> > > +
> > > + Any RID r in the interval [rid-base, rid-base + length) is associated with
> > > + the listed msi-controller, with the msi-specifier (r - rid-base + msi-base).
> > > +
> > > +- msi-map-mask: A mask to be applied to each Requester ID prior to being mapped
> > > + to an msi-specifier per the msi-map property.
> >
> > Can we extend the msi-map-mask definition to say: "A mask value of 0x0 is valid
> > and indicates that no RIDs are _currently_ mapped to any msi-specifier."
>
> That would break a valid case of the mask being all zeroes.
>
> Consider the case that all RIDs get mapped to a single msi-specifier;
> the obvious way to write that is:
>
> msi-map-mask = <0x0000>;
> msi-map = <0x0000 &msi (msi-specifier) 1>;
>
> In this case all RIDS are always mapped to the single msi-specifier.
Does it really break that case?
We could have this (your example):
msi-map-mask = <0x0000>;
msi-map = <0x0000 &msi 7 1>; // map all RIDs to msi-spec 7
Or, my example:
msi-map-mask = <0x0000>;
msi-map = <0x0000 &msi 7 4>; // all RIDs map to any of msi-spec 7,8,9,10
> > We have an SoC with a programmable hardware table in the PCI controller that maps
> > requester ID to stream ID, so the overall msi-map (and iommu-map) definition fit
> > into that scheme. But, we would like to be able make the RID->stream-ID mapping
> > decision _lazily_, in Linux, based on actual usage of PCI devices.
>
> Dynamically programming the mapping is at odds to this binding. I don't
> see how that can fit.
My example above obviously doesn't make sense for a static
binding, but can't we allow both a mask value of 0x0 and
a length > 1 at the same time.
> Why can the RID->SID mapping not be statically configured prior to
> entering the OS?
The problem for us is the limited number of SIDs available on our SoC. We
have an SMMU-500 with 128 SIDs / 64-contexts total. An SR-IOV card could
enable VFs dynamically and suddenly there are 64 new RIDs on a PCI
bus. There are 4 PCI controllers and firmware doesn't know what
might be enabled on which bus. We run out of available SIDs.
In that example we want all RIDs mapped to one SID by default, but want
the option of setting our dynamic RID->SID table for a situation where
say someone assigns a VF to a KVM VM and thus needs an SID to use
for SMMU mappings.
I think the binding can work for the dynamic case as long as we allow
the example I showed above. So, I would propose changing
my proposed text to:
"A mask value of 0x0 is valid and indicates that all RIDS map to
the specified msi-specifier(s). If the mask is 0x0 and length > 1
it indicates that all RIDs map to any of the msi-specifiers with
the actual mapping left unspecified by the msi-map property."
Thanks,
Stuart
Hi Mark,
Thanks for the response. Please find my comments inline.
Regards
Varun
> -----Original Message-----
> From: Mark Rutland [mailto:[email protected]]
> Sent: Thursday, August 06, 2015 11:09 PM
> To: Sethi Varun-B16395
> Cc: [email protected]; Lorenzo Pieralisi; [email protected]; Marc
> Zyngier; Will Deacon; [email protected];
> [email protected]; [email protected];
> [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; Yoder Stuart-B08248
> Subject: Re: [PATCH 2/3] Docs: dt: Add PCI MSI map bindings
>
> On Wed, Aug 05, 2015 at 05:39:33PM +0100, Varun Sethi wrote:
> > Hi Mark
> > Thanks for the patch. Please find my comment inline.
> >
> > Regards
> > Varun
> >
> > > -----Original Message-----
> > > From: [email protected] [mailto:iommu-
> > > [email protected]] On Behalf Of Mark Rutland
> > > Sent: Thursday, July 23, 2015 10:23 PM
> > > To: [email protected]
> > > Cc: Mark Rutland; [email protected]; [email protected];
> > > [email protected]; [email protected]; linux-
> > > [email protected]; [email protected];
> > > [email protected] foundation.org;
> > > [email protected];
> > > [email protected]; [email protected];
> > > [email protected]; [email protected];
> > > [email protected]
> > > Subject: [PATCH 2/3] Docs: dt: Add PCI MSI map bindings
> > >
> > > Currently msi-parent is used by a few bindings to describe the
> > > relationship between a PCI root complex and a single MSI controller,
> > > but this property does not have a generic binding document.
> > >
> > > Additionally, msi-parent is insufficient to describe more complex
> > > relationships between MSI controllers and devices under a root
> > > complex, where devices may be able to target multiple MSI
> > > controllers, or where MSI controllers use (non-probeable) sideband
> information to distinguish devices.
> > >
> > > This patch adds a generic binding for mapping PCI devices to MSI
> controllers.
> > > This document covers msi-parent, and a new msi-map property
> > > (specific to
> > > PCI*) which may be used to map devices (identified by their
> > > Requester ID) to sideband data for each MSI controller that they may
> target.
> > >
> > > Signed-off-by: Mark Rutland <[email protected]>
> > > ---
> > > Documentation/devicetree/bindings/pci/pci-msi.txt | 220
> > > ++++++++++++++++++++++
> > > 1 file changed, 220 insertions(+)
> > > create mode 100644
> > > Documentation/devicetree/bindings/pci/pci-msi.txt
> > >
> > > diff --git a/Documentation/devicetree/bindings/pci/pci-msi.txt
> > > b/Documentation/devicetree/bindings/pci/pci-msi.txt
> > > new file mode 100644
> > > index 0000000..9b3cc81
> > > --- /dev/null
> > > +++ b/Documentation/devicetree/bindings/pci/pci-msi.txt
> > > @@ -0,0 +1,220 @@
> > > +This document describes the generic device tree binding for
> > > +describing the relationship between PCI devices and MSI controllers.
> > > +
> > > +Each PCI device under a root complex is uniquely identified by its
> > > +Requester ID (AKA RID). A Requester ID is a triplet of a Bus
> > > +number, Device number, and Function number.
> > > +
> > > +For the purpose of this document, when treated as a numeric value,
> > > +a RID is formatted such that:
> > > +
> > > +* Bits [15:8] are the Bus number.
> > > +* Bits [7:3] are the Device number.
> > > +* Bits [2:0] are the Function number.
> > > +* Any other bits required for padding must be zero.
> > > +
> > > +MSIs may be distinguished in part through the use of sideband data
> > > +accompanying writes. In the case of PCI devices, this sideband data
> > > +may be derived from the Requester ID. A mechanism is required to
> > > +associate a device with both the MSI controllers it can address,
> > > +and the sideband data that will be associated with its writes to those
> controllers.
> > > +
> > > +For generic MSI bindings, see
> > > +Documentation/devicetree/bindings/interrupt-controller/msi.txt.
> > > +
> > > +
> > > +PCI root complex
> > > +================
> > > +
> > > +Optional properties
> > > +-------------------
> > > +
> > > +- msi-map: Maps a Requester ID to an MSI controller and associated
> > > + msi-specifier data. The property is an arbitrary number of tuples
> > > +of
> > > + (rid-base,msi-controller,msi-base,length), where:
> > [varun] How would we account for hot plug PCI devices and SR-IOV use
> cases, with the rid base and length?
>
> For hotplug, you simply need the mapping from RID to msi-specifier to be
> defined in advance in the DT, for the set of RIDs that could possibly occur.
>
> For SR-IOV, are you asking about ARI? I should update the description of the
> RID to describe that for ARI it has the format:
>
> * Bits [15:8] are the Bus number
> * Bits [7:0] are the Identifier
>
> Other than that, the handling would be identical to the non-ARI case.
>
> What else am I missing?
>
> > How do we take in to account for a PCIe bridge, while setting up the
> requestor ID base and length?
>
> I'm not sure I follow the question. I don't see why this is any different to any
> other requester ID.
>
> What do you see as being the problem for this case?
[varun] Would the boot loader be responsible for the PCI bus probe and setting up of the requestor id information in the device tree. Also, during bus probe the boot loader would understand the topology, and handle cases like non transparent host bridge and non standard devices (e.g. require special handling based on quirks).
Also, the msi identifier would typically be the stream ID provided by the device. So, this would be limited by the number of SMRs, right? Considering that the each device can have its specific IOMMU domain, which would also require mapping for MSI access, we may also be restricted by the number of context banks. So, we may be able to support a restricted number of msi identifiers.
We already have the PCI bus probe in Linux. If my assumption about the boot loader doing the probe is correct, why do we want to duplicate the work. Also, can we just provide a list of possible MSI identifiers to Linux for each PCIe controller. The IOMMU driver in the Linux kernel can interface with PCIe controller for setting up "requestor id"->"msi identifier" translation.
On Wed, Aug 05, 2015 at 05:51:20PM +0100, Mark Rutland wrote:
> Rob,
>
> Do you have any objections to this, or are you happy to take this patch?
>
> There's a user of this binding (the GICv3 ITS) queued for v4.3 already in
> the tip tree, so either we either need to be ok with this binding or we
> need to rework that before v4.3.
Sorry to ping, but are you happy to take this? Marc's replied and
provided his ack.
I'm happy to resend if you prefer?
Thanks,
Mark.
> Marc, can I take it from your use of the binding that you are happy to
> provide your ack?
>
> Thanks,
> Mark.
>
> On Thu, Jul 23, 2015 at 05:52:43PM +0100, Mark Rutland wrote:
> > Currently msi-parent is used in a couple of drivers despite being fairly
> > underspecified. This patch adds a generic binding for MSIs (including
> > the existing msi-parent property) enabling the description of platform
> > devices capable of using MSIs.
> >
> > While MSIs are primarily distinguished by doorbell and payload, some MSI
> > controllers (e.g. the GICv3 ITS) also use side-band information
> > accompanying the write to identify the master which originated the MSI,
> > to allow for sandboxing. This sideband information is non-probeable and
> > needs to be described in the DT. Other MSI controllers may have
> > additional configuration details which need to be described per-master.
> >
> > This patch adds a generic msi-parent binding document, extending the
> > de-facto standard with a new (optional) #msi-cells which can be used to
> > express any per-master configuration and/or sideband data. This is
> > sufficient to describe non-hotpluggable devices.
> >
> > For busses where sideband data may be derived from some bus-specific
> > master ID scheme, other properties will be required to describe the
> > mapping.
> >
> > Signed-off-by: Mark Rutland <[email protected]>
> > ---
> > .../bindings/interrupt-controller/msi.txt | 135 +++++++++++++++++++++
> > 1 file changed, 135 insertions(+)
> > create mode 100644 Documentation/devicetree/bindings/interrupt-controller/msi.txt
> >
> > diff --git a/Documentation/devicetree/bindings/interrupt-controller/msi.txt b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
> > new file mode 100644
> > index 0000000..c60c034
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
> > @@ -0,0 +1,135 @@
> > +This document describes the generic device tree binding for MSI controllers and
> > +their master(s).
> > +
> > +Message Signaled Interrupts (MSIs) are a class of interrupts generated by a
> > +write to an MMIO address.
> > +
> > +MSIs were originally specified by PCI (and are used with PCIe), but may also be
> > +used with other busses, and hence a mechanism is required to relate devices on
> > +those busses to the MSI controllers which they are capable of using,
> > +potentially including additional information.
> > +
> > +MSIs are distinguished by some combination of:
> > +
> > +- The doorbell (the MMIO address written to).
> > +
> > + Devices may be configured by software to write to arbitrary doorbells which
> > + they can address. An MSI controller may feature a number of doorbells.
> > +
> > +- The payload (the value written to the doorbell).
> > +
> > + Devices may be configured to write an arbitrary payload chosen by software.
> > + MSI controllers may have restrictions on permitted payloads.
> > +
> > +- Sideband information accompanying the write.
> > +
> > + Typically this is neither configurable nor probeable, and depends on the path
> > + taken through the memory system (i.e. it is a property of the combination of
> > + MSI controller and device rather than a property of either in isolation).
> > +
> > +
> > +MSI controllers:
> > +================
> > +
> > +An MSI controller signals interrupts to a CPU when a write is made to an MMIO
> > +address by some master. An MSI controller may feature a number of doorbells.
> > +
> > +Required properties:
> > +--------------------
> > +
> > +- msi-controller: Identifies the node as an MSI controller.
> > +
> > +Optional properties:
> > +--------------------
> > +
> > +- #msi-cells: The number of cells in an msi-specifier, required if not zero.
> > +
> > + Typically this will encode information related to sideband data, and will
> > + not encode doorbells or payloads as these can be configured dynamically.
> > +
> > + The meaning of the msi-specifier is defined by the device tree binding of
> > + the specific MSI controller.
> > +
> > +
> > +MSI clients
> > +===========
> > +
> > +MSI clients are devices which generate MSIs. For each MSI they wish to
> > +generate, the doorbell and payload may be configured, though sideband
> > +information may not be configurable.
> > +
> > +Required properties:
> > +--------------------
> > +
> > +- msi-parent: A list of phandle + msi-specifier pairs, one for each MSI
> > + controller which the device is capable of using.
> > +
> > + This property is unordered, and MSIs may be allocated from any combination of
> > + MSI controllers listed in the msi-parent property.
> > +
> > + If a device has restrictions on the allocation of MSIs, these restrictions
> > + must be described with additional properties.
> > +
> > + When #msi-cells is non-zero, busses with an msi-parent will require
> > + additional properties to describe the relationship between devices on the bus
> > + and the set of MSIs they can potentially generate.
> > +
> > +
> > +Example
> > +=======
> > +
> > +/ {
> > + #address-cells = <1>;
> > + #size-cells = <1>;
> > +
> > + msi_a: msi-controller@a {
> > + reg = <0xa 0xf00>;
> > + compatible = "vendor-a,some-controller";
> > + msi-controller;
> > + /* No sideband data, so #msi-cells omitted */
> > + };
> > +
> > + msi_b: msi-controller@b {
> > + reg = <0xb 0xf00>;
> > + compatible = "vendor-b,another-controller";
> > + msi-controller;
> > + /* Each device has some unique ID */
> > + #msi-cells = <1>;
> > + };
> > +
> > + msi_c: msi-controller@c {
> > + reg = <0xb 0xf00>;
> > + compatible = "vendor-b,another-controller";
> > + msi-controller;
> > + /* Each device has some unique ID */
> > + #msi-cells = <1>;
> > + };
> > +
> > + dev@0 {
> > + reg = <0x0 0xf00>;
> > + compatible = "vendor-c,some-device";
> > +
> > + /* Can only generate MSIs to msi_a */
> > + msi-parent = <&msi_a>;
> > + };
> > +
> > + dev@1 {
> > + reg = <0x1 0xf00>;
> > + compatible = "vendor-c,some-device";
> > +
> > + /*
> > + * Can generate MSIs to either A or B.
> > + */
> > + msi-parent = <&msi_a>, <&msi_b 0x17>;
> > + };
> > +
> > + dev@2 {
> > + reg = <0x2 0xf00>;
> > + compatible = "vendor-c,some-device";
> > + /*
> > + * Has different IDs at each MSI controller.
> > + * Can generate MSIs to all of the MSI controllers.
> > + */
> > + msi-parent = <&msi_a>, <&msi_b 0x17>, <&msi_c 0x53>;
> > + };
> > +};
> > --
> > 1.9.1
> >
> --
> To unsubscribe from this list: send the line "unsubscribe devicetree" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
On Mon, Aug 24, 2015 at 5:17 AM, Mark Rutland <[email protected]> wrote:
> On Wed, Aug 05, 2015 at 05:51:20PM +0100, Mark Rutland wrote:
>> Rob,
>>
>> Do you have any objections to this, or are you happy to take this patch?
>>
>> There's a user of this binding (the GICv3 ITS) queued for v4.3 already in
>> the tip tree, so either we either need to be ok with this binding or we
>> need to rework that before v4.3.
>
> Sorry to ping, but are you happy to take this? Marc's replied and
> provided his ack.
Sorry. Looks fine to me, so I'll apply.
> I'm happy to resend if you prefer?
What about patch 2 and 3? I can't find patch 3 in my mail.
Rob
>
> Thanks,
> Mark.
>
>> Marc, can I take it from your use of the binding that you are happy to
>> provide your ack?
>>
>> Thanks,
>> Mark.
>>
>> On Thu, Jul 23, 2015 at 05:52:43PM +0100, Mark Rutland wrote:
>> > Currently msi-parent is used in a couple of drivers despite being fairly
>> > underspecified. This patch adds a generic binding for MSIs (including
>> > the existing msi-parent property) enabling the description of platform
>> > devices capable of using MSIs.
>> >
>> > While MSIs are primarily distinguished by doorbell and payload, some MSI
>> > controllers (e.g. the GICv3 ITS) also use side-band information
>> > accompanying the write to identify the master which originated the MSI,
>> > to allow for sandboxing. This sideband information is non-probeable and
>> > needs to be described in the DT. Other MSI controllers may have
>> > additional configuration details which need to be described per-master.
>> >
>> > This patch adds a generic msi-parent binding document, extending the
>> > de-facto standard with a new (optional) #msi-cells which can be used to
>> > express any per-master configuration and/or sideband data. This is
>> > sufficient to describe non-hotpluggable devices.
>> >
>> > For busses where sideband data may be derived from some bus-specific
>> > master ID scheme, other properties will be required to describe the
>> > mapping.
>> >
>> > Signed-off-by: Mark Rutland <[email protected]>
>> > ---
>> > .../bindings/interrupt-controller/msi.txt | 135 +++++++++++++++++++++
>> > 1 file changed, 135 insertions(+)
>> > create mode 100644 Documentation/devicetree/bindings/interrupt-controller/msi.txt
>> >
>> > diff --git a/Documentation/devicetree/bindings/interrupt-controller/msi.txt b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
>> > new file mode 100644
>> > index 0000000..c60c034
>> > --- /dev/null
>> > +++ b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
>> > @@ -0,0 +1,135 @@
>> > +This document describes the generic device tree binding for MSI controllers and
>> > +their master(s).
>> > +
>> > +Message Signaled Interrupts (MSIs) are a class of interrupts generated by a
>> > +write to an MMIO address.
>> > +
>> > +MSIs were originally specified by PCI (and are used with PCIe), but may also be
>> > +used with other busses, and hence a mechanism is required to relate devices on
>> > +those busses to the MSI controllers which they are capable of using,
>> > +potentially including additional information.
>> > +
>> > +MSIs are distinguished by some combination of:
>> > +
>> > +- The doorbell (the MMIO address written to).
>> > +
>> > + Devices may be configured by software to write to arbitrary doorbells which
>> > + they can address. An MSI controller may feature a number of doorbells.
>> > +
>> > +- The payload (the value written to the doorbell).
>> > +
>> > + Devices may be configured to write an arbitrary payload chosen by software.
>> > + MSI controllers may have restrictions on permitted payloads.
>> > +
>> > +- Sideband information accompanying the write.
>> > +
>> > + Typically this is neither configurable nor probeable, and depends on the path
>> > + taken through the memory system (i.e. it is a property of the combination of
>> > + MSI controller and device rather than a property of either in isolation).
>> > +
>> > +
>> > +MSI controllers:
>> > +================
>> > +
>> > +An MSI controller signals interrupts to a CPU when a write is made to an MMIO
>> > +address by some master. An MSI controller may feature a number of doorbells.
>> > +
>> > +Required properties:
>> > +--------------------
>> > +
>> > +- msi-controller: Identifies the node as an MSI controller.
>> > +
>> > +Optional properties:
>> > +--------------------
>> > +
>> > +- #msi-cells: The number of cells in an msi-specifier, required if not zero.
>> > +
>> > + Typically this will encode information related to sideband data, and will
>> > + not encode doorbells or payloads as these can be configured dynamically.
>> > +
>> > + The meaning of the msi-specifier is defined by the device tree binding of
>> > + the specific MSI controller.
>> > +
>> > +
>> > +MSI clients
>> > +===========
>> > +
>> > +MSI clients are devices which generate MSIs. For each MSI they wish to
>> > +generate, the doorbell and payload may be configured, though sideband
>> > +information may not be configurable.
>> > +
>> > +Required properties:
>> > +--------------------
>> > +
>> > +- msi-parent: A list of phandle + msi-specifier pairs, one for each MSI
>> > + controller which the device is capable of using.
>> > +
>> > + This property is unordered, and MSIs may be allocated from any combination of
>> > + MSI controllers listed in the msi-parent property.
>> > +
>> > + If a device has restrictions on the allocation of MSIs, these restrictions
>> > + must be described with additional properties.
>> > +
>> > + When #msi-cells is non-zero, busses with an msi-parent will require
>> > + additional properties to describe the relationship between devices on the bus
>> > + and the set of MSIs they can potentially generate.
>> > +
>> > +
>> > +Example
>> > +=======
>> > +
>> > +/ {
>> > + #address-cells = <1>;
>> > + #size-cells = <1>;
>> > +
>> > + msi_a: msi-controller@a {
>> > + reg = <0xa 0xf00>;
>> > + compatible = "vendor-a,some-controller";
>> > + msi-controller;
>> > + /* No sideband data, so #msi-cells omitted */
>> > + };
>> > +
>> > + msi_b: msi-controller@b {
>> > + reg = <0xb 0xf00>;
>> > + compatible = "vendor-b,another-controller";
>> > + msi-controller;
>> > + /* Each device has some unique ID */
>> > + #msi-cells = <1>;
>> > + };
>> > +
>> > + msi_c: msi-controller@c {
>> > + reg = <0xb 0xf00>;
>> > + compatible = "vendor-b,another-controller";
>> > + msi-controller;
>> > + /* Each device has some unique ID */
>> > + #msi-cells = <1>;
>> > + };
>> > +
>> > + dev@0 {
>> > + reg = <0x0 0xf00>;
>> > + compatible = "vendor-c,some-device";
>> > +
>> > + /* Can only generate MSIs to msi_a */
>> > + msi-parent = <&msi_a>;
>> > + };
>> > +
>> > + dev@1 {
>> > + reg = <0x1 0xf00>;
>> > + compatible = "vendor-c,some-device";
>> > +
>> > + /*
>> > + * Can generate MSIs to either A or B.
>> > + */
>> > + msi-parent = <&msi_a>, <&msi_b 0x17>;
>> > + };
>> > +
>> > + dev@2 {
>> > + reg = <0x2 0xf00>;
>> > + compatible = "vendor-c,some-device";
>> > + /*
>> > + * Has different IDs at each MSI controller.
>> > + * Can generate MSIs to all of the MSI controllers.
>> > + */
>> > + msi-parent = <&msi_a>, <&msi_b 0x17>, <&msi_c 0x53>;
>> > + };
>> > +};
>> > --
>> > 1.9.1
>> >
>> --
>> To unsubscribe from this list: send the line "unsubscribe devicetree" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
On Mon, Aug 24, 2015 at 02:37:59PM +0100, Rob Herring wrote:
> On Mon, Aug 24, 2015 at 5:17 AM, Mark Rutland <[email protected]> wrote:
> > On Wed, Aug 05, 2015 at 05:51:20PM +0100, Mark Rutland wrote:
> >> Rob,
> >>
> >> Do you have any objections to this, or are you happy to take this patch?
> >>
> >> There's a user of this binding (the GICv3 ITS) queued for v4.3 already in
> >> the tip tree, so either we either need to be ok with this binding or we
> >> need to rework that before v4.3.
> >
> > Sorry to ping, but are you happy to take this? Marc's replied and
> > provided his ack.
>
> Sorry. Looks fine to me, so I'll apply.
Cheers!
> What about patch 2 and 3? I can't find patch 3 in my mail.
Those should wait for now.
We don't yet have a user and there's ongoing discussion w.r.t. the PCI
parts that I expect will pick up again as people return from LPC.
Thanks,
Mark.