On Fri, May 30, 2014 at 09:49:59PM +0200, Arnd Bergmann wrote:
> On Friday 30 May 2014 14:31:55 Rob Herring wrote:
> > On Fri, May 30, 2014 at 2:06 PM, Arnd Bergmann <[email protected]> wrote:
> > > On Friday 30 May 2014 08:16:05 Rob Herring wrote:
> > >> On Fri, May 23, 2014 at 3:33 PM, Thierry Reding
> > >> <[email protected]> wrote:
> > >> > From: Thierry Reding <[email protected]>
> > >> > +IOMMU master node:
> > >> > +==================
> > >> > +
> > >> > +Devices that access memory through an IOMMU are called masters. A device can
> > >> > +have multiple master interfaces (to one or more IOMMU devices).
> > >> > +
> > >> > +Required properties:
> > >> > +--------------------
> > >> > +- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
> > >> > + master interfaces of the device. One entry in the list describes one master
> > >> > + interface of the device.
> > >> > +
> > >> > +When an "iommus" property is specified in a device tree node, the IOMMU will
> > >> > +be used for address translation. If a "dma-ranges" property exists in the
> > >> > +device's parent node it will be ignored. An exception to this rule is if the
> > >> > +referenced IOMMU is disabled, in which case the "dma-ranges" property of the
> > >> > +parent shall take effect.
> > >>
> > >> Just thinking out loud, could you have dma-ranges in the iommu node
> > >> for the case when the iommu is enabled rather than putting the DMA
> > >> window information into the iommus property?
> > >>
> > >> This would probably mean that you need both #iommu-cells and #address-cells.
> > >
> > > The reason for doing like this was that you may need a different window
> > > for each device, while there can only be one dma-ranges property in
> > > an iommu node.
> >
> > My suggestion was that you also put the IDs in the dma-ranges as the
> > first cell much as ranges for PCI encodes other information in the
> > first cell. Then you can have an entry for each ID. The downside is
> > another special case like PCI.
> >
> > The argument for using #address-cells and #size-cells seems to be to
> > align with how ranges work. If that's the route we want to go, then I
> > say we should not stop there and actually use dma-ranges as well.
> > Otherwise, I don't see the advantage over #iommu-cells.
>
> I can see how dma-ranges in bus nodes work, it just doesn't seem to
> have any reasonable meaning in the iommu node itself.
dma-ranges defines a static mapping for mastering through the bus node.
The whole point of an IOMMU is that it maps dynamically, so I agree:
I'm unclear on what dma-ranges should mean in the IOMMU node itself
(if anything).
>
> > > I don't understand the problem. If you have stream IDs 0 through 7,
> > > you would have
> > >
> > > master@a {
> > > ...
> > > iommus = <&smmu 0>;
> > > };
> > >
> > > master@b {
> > > ...
> > > iommus = <&smmu 1;
> > > };
> > >
> > > ...
> > >
> > > master@12 {
> > > ...
> > > iommus = <&smmu 7;
> > > };
> > >
> > > and you don't need a window at all. Why would you need a mask of
> > > some sort?
> >
> > 1 master with 7 IDs like this:
> >
> > master@a {
> > ...
> > iommus = <&smmu 0> <&smmu 1> <&smmu 2> <&smmu 3>
> > <&smmu 4> <&smmu 5> <&smmu 6> <&smmu 7>;
> > };
> >
> > If there was any sort of window, then it is almost certainly the same
> > window for each ID.
Do we have real examples of using a window *and* an ID? I thought the
windowed-IOMMU concept was partly a way of encoding the ID in some
real address bits on the bus. If you're doing that, it seems less likely
that there is a true "ID" as such (though it is possible).
> Ok, I see. In that case you'd probably want to have #address-cells = <1>
> and #size-cells = <1> and give a range of IDs like
>
> iommus = <&smmu 0 8>;
>
> Do you think that ranges can have a meaningful definition with the ARM
> SMMU stream IDs?
In the strictest sense, no.
But for a large set of sane configurations, this probably works.
Small sets of randomly-assigned IDs can just be enumerated one by one.
We wouldn't be able to describe folding and bit shuffling, but we
probably don't want to encourage that anyway.
Cheers
---Dave
On Mon, Jun 02, 2014 at 11:41:04AM +0100, Dave Martin wrote:
> On Fri, May 30, 2014 at 09:49:59PM +0200, Arnd Bergmann wrote:
> > On Friday 30 May 2014 14:31:55 Rob Herring wrote:
> > > On Fri, May 30, 2014 at 2:06 PM, Arnd Bergmann <[email protected]> wrote:
> > > > On Friday 30 May 2014 08:16:05 Rob Herring wrote:
> > > >> On Fri, May 23, 2014 at 3:33 PM, Thierry Reding
> > > >> <[email protected]> wrote:
> > > >> > From: Thierry Reding <[email protected]>
> > > >> > +IOMMU master node:
> > > >> > +==================
> > > >> > +
> > > >> > +Devices that access memory through an IOMMU are called masters. A device can
> > > >> > +have multiple master interfaces (to one or more IOMMU devices).
> > > >> > +
> > > >> > +Required properties:
> > > >> > +--------------------
> > > >> > +- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
> > > >> > + master interfaces of the device. One entry in the list describes one master
> > > >> > + interface of the device.
> > > >> > +
> > > >> > +When an "iommus" property is specified in a device tree node, the IOMMU will
> > > >> > +be used for address translation. If a "dma-ranges" property exists in the
> > > >> > +device's parent node it will be ignored. An exception to this rule is if the
> > > >> > +referenced IOMMU is disabled, in which case the "dma-ranges" property of the
> > > >> > +parent shall take effect.
> > > >>
> > > >> Just thinking out loud, could you have dma-ranges in the iommu node
> > > >> for the case when the iommu is enabled rather than putting the DMA
> > > >> window information into the iommus property?
> > > >>
> > > >> This would probably mean that you need both #iommu-cells and #address-cells.
> > > >
> > > > The reason for doing like this was that you may need a different window
> > > > for each device, while there can only be one dma-ranges property in
> > > > an iommu node.
> > >
> > > My suggestion was that you also put the IDs in the dma-ranges as the
> > > first cell much as ranges for PCI encodes other information in the
> > > first cell. Then you can have an entry for each ID. The downside is
> > > another special case like PCI.
> > >
> > > The argument for using #address-cells and #size-cells seems to be to
> > > align with how ranges work. If that's the route we want to go, then I
> > > say we should not stop there and actually use dma-ranges as well.
> > > Otherwise, I don't see the advantage over #iommu-cells.
> >
> > I can see how dma-ranges in bus nodes work, it just doesn't seem to
> > have any reasonable meaning in the iommu node itself.
>
> dma-ranges defines a static mapping for mastering through the bus node.
>
> The whole point of an IOMMU is that it maps dynamically, so I agree:
> I'm unclear on what dma-ranges should mean in the IOMMU node itself
> (if anything).
>
> >
> > > > I don't understand the problem. If you have stream IDs 0 through 7,
> > > > you would have
> > > >
> > > > master@a {
> > > > ...
> > > > iommus = <&smmu 0>;
> > > > };
> > > >
> > > > master@b {
> > > > ...
> > > > iommus = <&smmu 1;
> > > > };
> > > >
> > > > ...
> > > >
> > > > master@12 {
> > > > ...
> > > > iommus = <&smmu 7;
> > > > };
> > > >
> > > > and you don't need a window at all. Why would you need a mask of
> > > > some sort?
> > >
> > > 1 master with 7 IDs like this:
> > >
> > > master@a {
> > > ...
> > > iommus = <&smmu 0> <&smmu 1> <&smmu 2> <&smmu 3>
> > > <&smmu 4> <&smmu 5> <&smmu 6> <&smmu 7>;
> > > };
> > >
> > > If there was any sort of window, then it is almost certainly the same
> > > window for each ID.
>
> Do we have real examples of using a window *and* an ID? I thought the
> windowed-IOMMU concept was partly a way of encoding the ID in some
> real address bits on the bus. If you're doing that, it seems less likely
> that there is a true "ID" as such (though it is possible).
>
> > Ok, I see. In that case you'd probably want to have #address-cells = <1>
> > and #size-cells = <1> and give a range of IDs like
> >
> > iommus = <&smmu 0 8>;
> >
> > Do you think that ranges can have a meaningful definition with the ARM
> > SMMU stream IDs?
>
> In the strictest sense, no.
>
> But for a large set of sane configurations, this probably works.
>
> Small sets of randomly-assigned IDs can just be enumerated one by one.
>
> We wouldn't be able to describe folding and bit shuffling, but we
> probably don't want to encourage that anyway.
I'm having some difficulty understanding this. You make it sound like
there's a fairly arbitrary number of IDs that the SMMU can handle. So
how is the mapping to devices defined? If you say encourage that does
make it sound like the assignment of IDs is purely defined by some
mechanism in software rather than in hardware. Or they are more or less
randomly picked by someone. If that's the case, is that not something
that should be dynamically allocated by the kernel rather than put into
the device tree?
Maybe in the above case all you really need to know is how many IDs a
device needs to be assigned so that they can be properly allocated,
rather than the device exactly specifying which ones.
Thierry
On Wed, Jun 04, 2014 at 03:35:10PM +0100, Thierry Reding wrote:
> On Mon, Jun 02, 2014 at 11:41:04AM +0100, Dave Martin wrote:
> > In the strictest sense, no.
> >
> > But for a large set of sane configurations, this probably works.
> >
> > Small sets of randomly-assigned IDs can just be enumerated one by one.
> >
> > We wouldn't be able to describe folding and bit shuffling, but we
> > probably don't want to encourage that anyway.
>
> I'm having some difficulty understanding this. You make it sound like
> there's a fairly arbitrary number of IDs that the SMMU can handle. So
> how is the mapping to devices defined? If you say encourage that does
> make it sound like the assignment of IDs is purely defined by some
> mechanism in software rather than in hardware. Or they are more or less
> randomly picked by someone. If that's the case, is that not something
> that should be dynamically allocated by the kernel rather than put into
> the device tree?
The set of StreamIDs that can be generated by a master is fixed in the
hardware. The SMMU can then be programmed to map these incoming IDs onto
a context ID (or a set of context IDs), which are the IDs used internally
by the SMMU to find the page tables etc.
The StreamID -> ContextID mapping is dynamic and controlled by software.
The Master -> StreamIDs mapping is fixed in the hardware.
Will
On Wed, Jun 04, 2014 at 05:41:32PM +0100, Will Deacon wrote:
> On Wed, Jun 04, 2014 at 03:35:10PM +0100, Thierry Reding wrote:
> > On Mon, Jun 02, 2014 at 11:41:04AM +0100, Dave Martin wrote:
> > > In the strictest sense, no.
> > >
> > > But for a large set of sane configurations, this probably works.
> > >
> > > Small sets of randomly-assigned IDs can just be enumerated one by one.
> > >
> > > We wouldn't be able to describe folding and bit shuffling, but we
> > > probably don't want to encourage that anyway.
> >
> > I'm having some difficulty understanding this. You make it sound like
> > there's a fairly arbitrary number of IDs that the SMMU can handle. So
> > how is the mapping to devices defined? If you say encourage that does
> > make it sound like the assignment of IDs is purely defined by some
> > mechanism in software rather than in hardware. Or they are more or less
> > randomly picked by someone. If that's the case, is that not something
> > that should be dynamically allocated by the kernel rather than put into
> > the device tree?
>
> The set of StreamIDs that can be generated by a master is fixed in the
> hardware. The SMMU can then be programmed to map these incoming IDs onto
> a context ID (or a set of context IDs), which are the IDs used internally
> by the SMMU to find the page tables etc.
>
> The StreamID -> ContextID mapping is dynamic and controlled by software.
> The Master -> StreamIDs mapping is fixed in the hardware.
Okay, that sounds similar to what the Tegra SMMU does. The naming is
slightly differently. "Master" is called "memory client", "StreamID"
would be "SWGROUP" and "ContextID" I guess would map to what's called an
"ASID" (Address Space ID) on Tegra.
I'm not sure about the amount of leverage that we have to encourage the
mappings that we think are easy to describe in DT and discourage those
that we don't want. Last time I checked we were still playing catch up
rather than being able to give recommendations to hardware engineers
about what will result in sane DT content and what won't.
Irrespective of the above, I still think that a generic binding based on
an #iommu-cells and iommus property gives us the most flexibility. For
easy cases, most of which are listed in the current proposal could
easily be dealt with in convenience helpers. For the more complex cases
drivers for those IOMMUs can define a specifier that make sense for the
mappings they need to be able to describe.
Also we're talking about a generic IOMMU binding here. That means that
if some hardware comes along that's not "generic" and doesn't fit into
the constraints set by this binding, then we still have the option of
defining a completely different one for that particular case.
Thierry
> -----Original Message-----
> From: [email protected] [mailto:iommu-
> [email protected]] On Behalf Of Will Deacon
> Sent: Wednesday, June 04, 2014 10:12 PM
> To: Thierry Reding
> Cc: Mark Rutland; [email protected]; linux-samsung-
> [email protected]; Pawel Moll; Arnd Bergmann; Ian Campbell; Grant
> Grundler; Stephen Warren; [email protected]; Marc Zyngier;
> Linux IOMMU; Rob Herring; Kumar Gala; [email protected]; Cho
> KyongHo; Dave P Martin; [email protected]
> Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
> bindings
>
> On Wed, Jun 04, 2014 at 03:35:10PM +0100, Thierry Reding wrote:
> > On Mon, Jun 02, 2014 at 11:41:04AM +0100, Dave Martin wrote:
> > > In the strictest sense, no.
> > >
> > > But for a large set of sane configurations, this probably works.
> > >
> > > Small sets of randomly-assigned IDs can just be enumerated one by
> one.
> > >
> > > We wouldn't be able to describe folding and bit shuffling, but we
> > > probably don't want to encourage that anyway.
> >
> > I'm having some difficulty understanding this. You make it sound like
> > there's a fairly arbitrary number of IDs that the SMMU can handle. So
> > how is the mapping to devices defined? If you say encourage that does
> > make it sound like the assignment of IDs is purely defined by some
> > mechanism in software rather than in hardware. Or they are more or
> > less randomly picked by someone. If that's the case, is that not
> > something that should be dynamically allocated by the kernel rather
> > than put into the device tree?
>
> The set of StreamIDs that can be generated by a master is fixed in the
> hardware. The SMMU can then be programmed to map these incoming IDs onto
> a context ID (or a set of context IDs), which are the IDs used internally
> by the SMMU to find the page tables etc.
>
> The StreamID -> ContextID mapping is dynamic and controlled by software.
> The Master -> StreamIDs mapping is fixed in the hardware.
The Master -> StreamIDs mapping may not always be fixed in the hardware. At, least in our case we plan to program these via software. PCI devices is one place where this mapping would have to be dynamic (based on the topology i.e. if the devices are connected to a bridge etc). Also, we have a hot plug device architecture where the stream ID is software programmable.
Other than that, based on the isolation requirements (iommu domain) software programmability offers greater flexibility.
-Varun
Hi Varun,
On Thu, Jun 05, 2014 at 08:10:19PM +0100, Varun Sethi wrote:
> > The set of StreamIDs that can be generated by a master is fixed in the
> > hardware. The SMMU can then be programmed to map these incoming IDs onto
> > a context ID (or a set of context IDs), which are the IDs used internally
> > by the SMMU to find the page tables etc.
> >
> > The StreamID -> ContextID mapping is dynamic and controlled by software.
> > The Master -> StreamIDs mapping is fixed in the hardware.
> The Master -> StreamIDs mapping may not always be fixed in the hardware.
> At, least in our case we plan to program these via software. PCI devices
> is one place where this mapping would have to be dynamic (based on the
> topology i.e. if the devices are connected to a bridge etc). Also, we have
> a hot plug device architecture where the stream ID is software
> programmable.
>
> Other than that, based on the isolation requirements (iommu domain)
> software programmability offers greater flexibility.
For the sake of consistency, I'd really like to treat the master ->
streamIDs mapping as fixed. If that means setting it once during boot in
your case, then so be it (ideally in the firmware). That way, we just
describe the fixed mapping to the kernel and the driver doesn't have to
worry about changing the mapping, especially given that that's likely to be
highly error-prone once the SMMU is in us.
Do you have use-cases where you really need to change these mappings
dynamically?
Will
> -----Original Message-----
> From: Will Deacon [mailto:[email protected]]
> Sent: Monday, June 16, 2014 10:28 AM
> To: Sethi Varun-B16395
> Cc: Thierry Reding; Mark Rutland; [email protected]; linux-
> [email protected]; Pawel Moll; Arnd Bergmann; Ian Campbell;
> Grant Grundler; Stephen Warren; [email protected]; Marc
> Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux-
> [email protected]; Cho KyongHo; Dave P Martin; linux-arm-
> [email protected]; Yoder Stuart-B08248
> Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
> bindings
>
> Hi Varun,
>
> On Thu, Jun 05, 2014 at 08:10:19PM +0100, Varun Sethi wrote:
> > > The set of StreamIDs that can be generated by a master is fixed in
> the
> > > hardware. The SMMU can then be programmed to map these incoming IDs
> onto
> > > a context ID (or a set of context IDs), which are the IDs used
> internally
> > > by the SMMU to find the page tables etc.
> > >
> > > The StreamID -> ContextID mapping is dynamic and controlled by
> software.
> > > The Master -> StreamIDs mapping is fixed in the hardware.
> > The Master -> StreamIDs mapping may not always be fixed in the
> hardware.
> > At, least in our case we plan to program these via software. PCI
> devices
> > is one place where this mapping would have to be dynamic (based on the
> > topology i.e. if the devices are connected to a bridge etc). Also, we
> have
> > a hot plug device architecture where the stream ID is software
> > programmable.
> >
> > Other than that, based on the isolation requirements (iommu domain)
> > software programmability offers greater flexibility.
>
> For the sake of consistency, I'd really like to treat the master ->
> streamIDs mapping as fixed. If that means setting it once during boot in
> your case, then so be it (ideally in the firmware). That way, we just
> describe the fixed mapping to the kernel and the driver doesn't have to
> worry about changing the mapping, especially given that that's likely to
> be
> highly error-prone once the SMMU is in us.
>
> Do you have use-cases where you really need to change these mappings
> dynamically?
Yes. In the case of a PCI bus-- you may not know in advance how many
PCI devices there are until you probe the bus. We have another FSL
proprietary bus we call the "fsl-mc" bus that is similar.
Another thing to consider-- starting with SMMUv2, as you know, there
is a new distributed architecture with multiple TBUs and a centralized
TCU that walks the SMMU page tables. So instead of sprinkling multiple
SMMUs all over an SoC you now have the option a 1 central TCU and sprinkling
multiple TBUs around. However, this means that the stream ID namespace
is now global and can be pretty limited. In the SMMU implementation we
have there are only 64 stream ID total for our Soc. But we have many more
masters than that.
So we look at stream IDs as really corresponding to an 'isolation context'
and not to a bus master. An isolation context is the domain you are
trying to isolate with the SMMU. Devices that all belong to the same
'isolation context' can share the same stream ID, since they share
the same domain and page tables.
So, perhaps by default some/most SMMU masters may have a default stream ID
of 0x0 that is used by the host...and that could be represented
statically in the device tree.
But, we absolutely will need to dynamically set new stream IDs
into masters when a new IOMMU 'domain' is created and devices
are added to it. All the devices in a domain will share
the same stream ID.
So whatever we do, let's please have an architecture flexible enough
to allow for this.
Thanks,
Stuart
Hi Stuart,
On Mon, Jun 16, 2014 at 05:56:32PM +0100, Stuart Yoder wrote:
> > Do you have use-cases where you really need to change these mappings
> > dynamically?
>
> Yes. In the case of a PCI bus-- you may not know in advance how many
> PCI devices there are until you probe the bus. We have another FSL
> proprietary bus we call the "fsl-mc" bus that is similar.
For that case, though, you could still describe an algorithmic
transformation from RequesterID to StreamID which corresponds to a fixed
mapping.
> Another thing to consider-- starting with SMMUv2, as you know, there
> is a new distributed architecture with multiple TBUs and a centralized
> TCU that walks the SMMU page tables. So instead of sprinkling multiple
> SMMUs all over an SoC you now have the option a 1 central TCU and sprinkling
> multiple TBUs around. However, this means that the stream ID namespace
> is now global and can be pretty limited. In the SMMU implementation we
> have there are only 64 stream ID total for our Soc. But we have many more
> masters than that.
>
> So we look at stream IDs as really corresponding to an 'isolation context'
> and not to a bus master. An isolation context is the domain you are
> trying to isolate with the SMMU. Devices that all belong to the same
> 'isolation context' can share the same stream ID, since they share
> the same domain and page tables.
Ok, this is more compelling.
> So, perhaps by default some/most SMMU masters may have a default stream ID
> of 0x0 that is used by the host...and that could be represented
> statically in the device tree.
>
> But, we absolutely will need to dynamically set new stream IDs
> into masters when a new IOMMU 'domain' is created and devices
> are added to it. All the devices in a domain will share
> the same stream ID.
>
> So whatever we do, let's please have an architecture flexible enough
> to allow for this.
What is the software interface to the logic that assigns the StreamIDs? Is
it part of the SMMU, or a separate device (or set of devices)?
Will
On Monday 16 June 2014 18:04:16 Will Deacon wrote:
>
> On Mon, Jun 16, 2014 at 05:56:32PM +0100, Stuart Yoder wrote:
> > > Do you have use-cases where you really need to change these mappings
> > > dynamically?
> >
> > Yes. In the case of a PCI bus-- you may not know in advance how many
> > PCI devices there are until you probe the bus. We have another FSL
> > proprietary bus we call the "fsl-mc" bus that is similar.
>
> For that case, though, you could still describe an algorithmic
> transformation from RequesterID to StreamID which corresponds to a fixed
> mapping.
It sounds to me like the best option here would be to have only the
RequesterID passed in the "iommus" property and have the StreamID
dynamically assigned. This would mean we treat the StreamID as
the context.
> > Another thing to consider-- starting with SMMUv2, as you know, there
> > is a new distributed architecture with multiple TBUs and a centralized
> > TCU that walks the SMMU page tables. So instead of sprinkling multiple
> > SMMUs all over an SoC you now have the option a 1 central TCU and sprinkling
> > multiple TBUs around. However, this means that the stream ID namespace
> > is now global and can be pretty limited. In the SMMU implementation we
> > have there are only 64 stream ID total for our Soc. But we have many more
> > masters than that.
> >
> > So we look at stream IDs as really corresponding to an 'isolation context'
> > and not to a bus master. An isolation context is the domain you are
> > trying to isolate with the SMMU. Devices that all belong to the same
> > 'isolation context' can share the same stream ID, since they share
> > the same domain and page tables.
>
> Ok, this is more compelling.
This also makes sense from the kernel's perspective: The shared Stream ID
is the one that is used by the dma-mapping API here, while all other
Stream IDs would only be used if you have things like PCI device assignment
or GPU contexts that map into IOMMU contexts using the Linux IOMMU
abstraction, which is a lower-level interface than the dma-mapping
abstraction.
Arnd
> -----Original Message-----
> From: Will Deacon [mailto:[email protected]]
> Sent: Monday, June 16, 2014 12:04 PM
> To: Yoder Stuart-B08248
> Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
> [email protected]; [email protected]; Pawel
> Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux-
> [email protected]; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar
> Gala; [email protected]; Cho KyongHo; Dave P Martin; linux-arm-
> [email protected]
> Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
> bindings
>
> Hi Stuart,
>
> On Mon, Jun 16, 2014 at 05:56:32PM +0100, Stuart Yoder wrote:
> > > Do you have use-cases where you really need to change these mappings
> > > dynamically?
> >
> > Yes. In the case of a PCI bus-- you may not know in advance how many
> > PCI devices there are until you probe the bus. We have another FSL
> > proprietary bus we call the "fsl-mc" bus that is similar.
>
> For that case, though, you could still describe an algorithmic
> transformation from RequesterID to StreamID which corresponds to a fixed
> mapping.
>
> > Another thing to consider-- starting with SMMUv2, as you know, there
> > is a new distributed architecture with multiple TBUs and a centralized
> > TCU that walks the SMMU page tables. So instead of sprinkling multiple
> > SMMUs all over an SoC you now have the option a 1 central TCU and
> sprinkling
> > multiple TBUs around. However, this means that the stream ID
> namespace
> > is now global and can be pretty limited. In the SMMU implementation we
> > have there are only 64 stream ID total for our Soc. But we have many
> more
> > masters than that.
> >
> > So we look at stream IDs as really corresponding to an 'isolation
> context'
> > and not to a bus master. An isolation context is the domain you are
> > trying to isolate with the SMMU. Devices that all belong to the same
> > 'isolation context' can share the same stream ID, since they share
> > the same domain and page tables.
>
> Ok, this is more compelling.
>
> > So, perhaps by default some/most SMMU masters may have a default stream
> ID
> > of 0x0 that is used by the host...and that could be represented
> > statically in the device tree.
> >
> > But, we absolutely will need to dynamically set new stream IDs
> > into masters when a new IOMMU 'domain' is created and devices
> > are added to it. All the devices in a domain will share
> > the same stream ID.
> >
> > So whatever we do, let's please have an architecture flexible enough
> > to allow for this.
>
> What is the software interface to the logic that assigns the StreamIDs?
> Is
> it part of the SMMU, or a separate device (or set of devices)?
For us at the hardware level there are a few different ways that the streamIDs
can be set. It is not part of the SMMU. In the cases where there is simply
1 device to 1 streamID (e.g. USB controller) there is an SoC register where
you just program the stream ID. In the case of PCI, our PCI controller
has a RequesterID-to-streamID table that you set via some PCI controller
registers.
The way we generally thought it would work was something like
this:
-u-boot/bootloader makes any static streamID allocation if needed,
sets a default streamID (e.g. 0x0) in device and expresses
that in the device tree
-device tree would express relationship between devices
(including bus controllers) and the SMMU through mmu-masters
property
-u-boot would express the range of unused (or used) streamIDs via a new
device tree property so the kernel SMMU driver knows what streamIDs are
free
-in the SMMU driver a different vendor specific 'add_device' callback
could be used to handle our special cases where we need to set/change
the stream ID for devices added to a domain
Thanks,
Stuart
> -----Original Message-----
> From: Yoder Stuart-B08248
> Sent: Tuesday, June 17, 2014 12:24 AM
> To: Will Deacon
> Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
> [email protected]; [email protected]; Pawel
> Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux-
> [email protected]; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar
> Gala; [email protected]; Cho KyongHo; Dave P Martin; linux-arm-
> [email protected]
> Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree
> bindings
>
>
>
> > -----Original Message-----
> > From: Will Deacon [mailto:[email protected]]
> > Sent: Monday, June 16, 2014 12:04 PM
> > To: Yoder Stuart-B08248
> > Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
> > [email protected]; [email protected]; Pawel
> > Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren;
> > linux- [email protected]; Marc Zyngier; Linux IOMMU; Rob Herring;
> > Kumar Gala; [email protected]; Cho KyongHo; Dave P Martin;
> > linux-arm- [email protected]
> > Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
> > bindings
> >
> > Hi Stuart,
> >
> > On Mon, Jun 16, 2014 at 05:56:32PM +0100, Stuart Yoder wrote:
> > > > Do you have use-cases where you really need to change these
> > > > mappings dynamically?
> > >
> > > Yes. In the case of a PCI bus-- you may not know in advance how many
> > > PCI devices there are until you probe the bus. We have another FSL
> > > proprietary bus we call the "fsl-mc" bus that is similar.
> >
> > For that case, though, you could still describe an algorithmic
> > transformation from RequesterID to StreamID which corresponds to a
> > fixed mapping.
> >
> > > Another thing to consider-- starting with SMMUv2, as you know, there
> > > is a new distributed architecture with multiple TBUs and a
> > > centralized TCU that walks the SMMU page tables. So instead of
> > > sprinkling multiple SMMUs all over an SoC you now have the option a
> > > 1 central TCU and
> > sprinkling
> > > multiple TBUs around. However, this means that the stream ID
> > namespace
> > > is now global and can be pretty limited. In the SMMU implementation
> > > we have there are only 64 stream ID total for our Soc. But we have
> > > many
> > more
> > > masters than that.
> > >
> > > So we look at stream IDs as really corresponding to an 'isolation
> > context'
> > > and not to a bus master. An isolation context is the domain you are
> > > trying to isolate with the SMMU. Devices that all belong to the
> > > same 'isolation context' can share the same stream ID, since they
> > > share the same domain and page tables.
> >
> > Ok, this is more compelling.
> >
> > > So, perhaps by default some/most SMMU masters may have a default
> > > stream
> > ID
> > > of 0x0 that is used by the host...and that could be represented
> > > statically in the device tree.
> > >
> > > But, we absolutely will need to dynamically set new stream IDs into
> > > masters when a new IOMMU 'domain' is created and devices
> > > are added to it. All the devices in a domain will share
> > > the same stream ID.
> > >
> > > So whatever we do, let's please have an architecture flexible enough
> > > to allow for this.
> >
> > What is the software interface to the logic that assigns the StreamIDs?
> > Is
> > it part of the SMMU, or a separate device (or set of devices)?
>
> For us at the hardware level there are a few different ways that the
> streamIDs can be set. It is not part of the SMMU. In the cases where
> there is simply
> 1 device to 1 streamID (e.g. USB controller) there is an SoC register
> where
> you just program the stream ID. In the case of PCI, our PCI controller
> has a RequesterID-to-streamID table that you set via some PCI controller
> registers.
>
> The way we generally thought it would work was something like
> this:
> -u-boot/bootloader makes any static streamID allocation if needed,
> sets a default streamID (e.g. 0x0) in device and expresses
> that in the device tree
> -device tree would express relationship between devices
> (including bus controllers) and the SMMU through mmu-masters
> property
> -u-boot would express the range of unused (or used) streamIDs via a
> new
> device tree property so the kernel SMMU driver knows what streamIDs
> are
> free
> -in the SMMU driver a different vendor specific 'add_device' callback
> could be used to handle our special cases where we need to set/change
> the stream ID for devices added to a domain
Another possibility, could be to program the stream Id in the device registers (reference for the stream ID register can be obtained from the device tree) during device attach. This could be relevant in case of VFIO, when we are assigning multiple devices to a single VM. All the devices can share the same stream ID.
-Varun
On Tue, Jun 17, 2014 at 11:26:48AM +0100, Varun Sethi wrote:
> > The way we generally thought it would work was something like
> > this:
> > -u-boot/bootloader makes any static streamID allocation if needed,
> > sets a default streamID (e.g. 0x0) in device and expresses
> > that in the device tree
> > -device tree would express relationship between devices
> > (including bus controllers) and the SMMU through mmu-masters
> > property
> > -u-boot would express the range of unused (or used) streamIDs via a
> > new
> > device tree property so the kernel SMMU driver knows what streamIDs
> > are
> > free
> > -in the SMMU driver a different vendor specific 'add_device' callback
> > could be used to handle our special cases where we need to set/change
> > the stream ID for devices added to a domain
>
> Another possibility, could be to program the stream Id in the device
> registers (reference for the stream ID register can be obtained from the
> device tree) during device attach. This could be relevant in case of VFIO,
> when we are assigning multiple devices to a single VM. All the devices can
> share the same stream ID.
I think for simple masters (i.e. those that have all their StreamIDs under
control of one driver), then setting something during attach (or add?)
based on the DT could work pretty well. The other case is when we have
masters behind a bridge (such as a PCI RC). In this case, it might actually
be better to ask the bridge for the IDs and let it sort out the allocation
itself. That would also move the RequesterID -> StreamID mapping out of
the SMMU code.
What do you think?
Will
> -----Original Message-----
> From: [email protected] [mailto:iommu-
> [email protected]] On Behalf Of Will Deacon
> Sent: Tuesday, June 17, 2014 4:13 PM
> To: Sethi Varun-B16395
> Cc: Mark Rutland; [email protected]; linux-samsung-
> [email protected]; Arnd Bergmann; Pawel Moll; Ian Campbell; Grant
> Grundler; Stephen Warren; Yoder Stuart-B08248; Rob Herring; linux-
> [email protected]; Marc Zyngier; Linux IOMMU; Thierry Reding; Kumar
> Gala; [email protected]; Cho KyongHo; Dave P Martin; linux-arm-
> [email protected]
> Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
> bindings
>
> On Tue, Jun 17, 2014 at 11:26:48AM +0100, Varun Sethi wrote:
> > > The way we generally thought it would work was something like
> > > this:
> > > -u-boot/bootloader makes any static streamID allocation if needed,
> > > sets a default streamID (e.g. 0x0) in device and expresses
> > > that in the device tree
> > > -device tree would express relationship between devices
> > > (including bus controllers) and the SMMU through mmu-masters
> > > property
> > > -u-boot would express the range of unused (or used) streamIDs via
> > > a new
> > > device tree property so the kernel SMMU driver knows what
> > > streamIDs are
> > > free
> > > -in the SMMU driver a different vendor specific 'add_device'
> callback
> > > could be used to handle our special cases where we need to
> set/change
> > > the stream ID for devices added to a domain
> >
> > Another possibility, could be to program the stream Id in the device
> > registers (reference for the stream ID register can be obtained from
> > the device tree) during device attach. This could be relevant in case
> > of VFIO, when we are assigning multiple devices to a single VM. All
> > the devices can share the same stream ID.
>
> I think for simple masters (i.e. those that have all their StreamIDs
> under control of one driver), then setting something during attach (or
> add?) based on the DT could work pretty well. The other case is when we
> have masters behind a bridge (such as a PCI RC). In this case, it might
> actually be better to ask the bridge for the IDs and let it sort out the
> allocation itself. That would also move the RequesterID -> StreamID
> mapping out of the SMMU code.
>
> What do you think?
The PCI bus iommu group creation code would be part of the SMMU driver (it is handled by other IOMMU drivers as well). My understanding is that there would be one is to one correspondence between the requestor ID and the iommu group. May be, we can have an API provided by the PCI bridge (architecture specific) for setting the stream ID.
-Varun
> -----Original Message-----
> From: Sethi Varun-B16395
> Sent: Tuesday, June 17, 2014 5:27 AM
> To: Yoder Stuart-B08248; Will Deacon
> Cc: Thierry Reding; Mark Rutland; [email protected]; linux-
> [email protected]; Pawel Moll; Arnd Bergmann; Ian Campbell;
> Grant Grundler; Stephen Warren; [email protected]; Marc
> Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux-
> [email protected]; Cho KyongHo; Dave P Martin; linux-arm-
> [email protected]
> Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree
> bindings
>
>
>
> > -----Original Message-----
> > From: Yoder Stuart-B08248
> > Sent: Tuesday, June 17, 2014 12:24 AM
> > To: Will Deacon
> > Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
> > [email protected]; [email protected]; Pawel
> > Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren;
> linux-
> > [email protected]; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar
> > Gala; [email protected]; Cho KyongHo; Dave P Martin; linux-
> arm-
> > [email protected]
> > Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree
> > bindings
> >
> >
> >
> > > -----Original Message-----
> > > From: Will Deacon [mailto:[email protected]]
> > > Sent: Monday, June 16, 2014 12:04 PM
> > > To: Yoder Stuart-B08248
> > > Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
> > > [email protected]; [email protected]; Pawel
> > > Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren;
> > > linux- [email protected]; Marc Zyngier; Linux IOMMU; Rob
> Herring;
> > > Kumar Gala; [email protected]; Cho KyongHo; Dave P Martin;
> > > linux-arm- [email protected]
> > > Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
> > > bindings
> > >
> > > Hi Stuart,
> > >
> > > On Mon, Jun 16, 2014 at 05:56:32PM +0100, Stuart Yoder wrote:
> > > > > Do you have use-cases where you really need to change these
> > > > > mappings dynamically?
> > > >
> > > > Yes. In the case of a PCI bus-- you may not know in advance how
> many
> > > > PCI devices there are until you probe the bus. We have another
> FSL
> > > > proprietary bus we call the "fsl-mc" bus that is similar.
> > >
> > > For that case, though, you could still describe an algorithmic
> > > transformation from RequesterID to StreamID which corresponds to a
> > > fixed mapping.
> > >
> > > > Another thing to consider-- starting with SMMUv2, as you know,
> there
> > > > is a new distributed architecture with multiple TBUs and a
> > > > centralized TCU that walks the SMMU page tables. So instead of
> > > > sprinkling multiple SMMUs all over an SoC you now have the option a
> > > > 1 central TCU and
> > > sprinkling
> > > > multiple TBUs around. However, this means that the stream ID
> > > namespace
> > > > is now global and can be pretty limited. In the SMMU
> implementation
> > > > we have there are only 64 stream ID total for our Soc. But we have
> > > > many
> > > more
> > > > masters than that.
> > > >
> > > > So we look at stream IDs as really corresponding to an 'isolation
> > > context'
> > > > and not to a bus master. An isolation context is the domain you
> are
> > > > trying to isolate with the SMMU. Devices that all belong to the
> > > > same 'isolation context' can share the same stream ID, since they
> > > > share the same domain and page tables.
> > >
> > > Ok, this is more compelling.
> > >
> > > > So, perhaps by default some/most SMMU masters may have a default
> > > > stream
> > > ID
> > > > of 0x0 that is used by the host...and that could be represented
> > > > statically in the device tree.
> > > >
> > > > But, we absolutely will need to dynamically set new stream IDs into
> > > > masters when a new IOMMU 'domain' is created and devices
> > > > are added to it. All the devices in a domain will share
> > > > the same stream ID.
> > > >
> > > > So whatever we do, let's please have an architecture flexible
> enough
> > > > to allow for this.
> > >
> > > What is the software interface to the logic that assigns the
> StreamIDs?
> > > Is
> > > it part of the SMMU, or a separate device (or set of devices)?
> >
> > For us at the hardware level there are a few different ways that the
> > streamIDs can be set. It is not part of the SMMU. In the cases where
> > there is simply
> > 1 device to 1 streamID (e.g. USB controller) there is an SoC register
> > where
> > you just program the stream ID. In the case of PCI, our PCI
> controller
> > has a RequesterID-to-streamID table that you set via some PCI
> controller
> > registers.
> >
> > The way we generally thought it would work was something like
> > this:
> > -u-boot/bootloader makes any static streamID allocation if needed,
> > sets a default streamID (e.g. 0x0) in device and expresses
> > that in the device tree
> > -device tree would express relationship between devices
> > (including bus controllers) and the SMMU through mmu-masters
> > property
> > -u-boot would express the range of unused (or used) streamIDs via a
> > new
> > device tree property so the kernel SMMU driver knows what streamIDs
> > are
> > free
> > -in the SMMU driver a different vendor specific 'add_device'
> callback
> > could be used to handle our special cases where we need to
> set/change
> > the stream ID for devices added to a domain
>
> Another possibility, could be to program the stream Id in the device
> registers (reference for the stream ID register can be obtained from the
> device tree) during device attach. This could be relevant in case of
> VFIO, when we are assigning multiple devices to a single VM. All the
> devices can share the same stream ID.
Actually, that is what I meant-- do the special case handling during
device "attach" (not add).
Stuart
> -----Original Message-----
> From: Sethi Varun-B16395
> Sent: Tuesday, June 17, 2014 6:22 AM
> To: Will Deacon
> Cc: Mark Rutland; [email protected]; linux-samsung-
> [email protected]; Arnd Bergmann; Pawel Moll; Ian Campbell; Grant
> Grundler; Stephen Warren; Yoder Stuart-B08248; Rob Herring; linux-
> [email protected]; Marc Zyngier; Linux IOMMU; Thierry Reding; Kumar
> Gala; [email protected]; Cho KyongHo; Dave P Martin; linux-arm-
> [email protected]
> Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree
> bindings
>
>
>
> > -----Original Message-----
> > From: [email protected] [mailto:iommu-
> > [email protected]] On Behalf Of Will Deacon
> > Sent: Tuesday, June 17, 2014 4:13 PM
> > To: Sethi Varun-B16395
> > Cc: Mark Rutland; [email protected]; linux-samsung-
> > [email protected]; Arnd Bergmann; Pawel Moll; Ian Campbell; Grant
> > Grundler; Stephen Warren; Yoder Stuart-B08248; Rob Herring; linux-
> > [email protected]; Marc Zyngier; Linux IOMMU; Thierry Reding;
> Kumar
> > Gala; [email protected]; Cho KyongHo; Dave P Martin; linux-
> arm-
> > [email protected]
> > Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
> > bindings
> >
> > On Tue, Jun 17, 2014 at 11:26:48AM +0100, Varun Sethi wrote:
> > > > The way we generally thought it would work was something like
> > > > this:
> > > > -u-boot/bootloader makes any static streamID allocation if
> needed,
> > > > sets a default streamID (e.g. 0x0) in device and expresses
> > > > that in the device tree
> > > > -device tree would express relationship between devices
> > > > (including bus controllers) and the SMMU through mmu-masters
> > > > property
> > > > -u-boot would express the range of unused (or used) streamIDs
> via
> > > > a new
> > > > device tree property so the kernel SMMU driver knows what
> > > > streamIDs are
> > > > free
> > > > -in the SMMU driver a different vendor specific 'add_device'
> > callback
> > > > could be used to handle our special cases where we need to
> > set/change
> > > > the stream ID for devices added to a domain
> > >
> > > Another possibility, could be to program the stream Id in the device
> > > registers (reference for the stream ID register can be obtained from
> > > the device tree) during device attach. This could be relevant in case
> > > of VFIO, when we are assigning multiple devices to a single VM. All
> > > the devices can share the same stream ID.
> >
> > I think for simple masters (i.e. those that have all their StreamIDs
> > under control of one driver), then setting something during attach (or
> > add?) based on the DT could work pretty well. The other case is when we
> > have masters behind a bridge (such as a PCI RC). In this case, it might
> > actually be better to ask the bridge for the IDs and let it sort out
> the
> > allocation itself. That would also move the RequesterID -> StreamID
> > mapping out of the SMMU code.
> >
> > What do you think?
> The PCI bus iommu group creation code would be part of the SMMU driver
> (it is handled by other IOMMU drivers as well). My understanding is that
> there would be one is to one correspondence between the requestor ID and
> the iommu group. May be, we can have an API provided by the PCI bridge
> (architecture specific) for setting the stream ID.
I think Will is suggesting something along those lines-- I think it's a
question of where the streamID allocation happens. You could
either do something like the following in the SMMU driver when attaching
a PCI device:
id = alloc_stream_id(...);
pci_set_streamid(pci-dev, id);
or
id = pci_get_streamid(pci-dev);
...i.e the PCI RC could allocate (from some TBD
allocator) and set the stream ID itself.
Not sure how big a deal it is to extend PCI RC interfaces for
something like that.
Stuart
On Tue, Jun 17, 2014 at 03:50:25PM +0100, Stuart Yoder wrote:
> > > I think for simple masters (i.e. those that have all their StreamIDs
> > > under control of one driver), then setting something during attach (or
> > > add?) based on the DT could work pretty well. The other case is when we
> > > have masters behind a bridge (such as a PCI RC). In this case, it might
> > > actually be better to ask the bridge for the IDs and let it sort out
> > the
> > > allocation itself. That would also move the RequesterID -> StreamID
> > > mapping out of the SMMU code.
> > >
> > > What do you think?
> > The PCI bus iommu group creation code would be part of the SMMU driver
> > (it is handled by other IOMMU drivers as well). My understanding is that
> > there would be one is to one correspondence between the requestor ID and
> > the iommu group. May be, we can have an API provided by the PCI bridge
> > (architecture specific) for setting the stream ID.
>
> I think Will is suggesting something along those lines-- I think it's a
> question of where the streamID allocation happens. You could
> either do something like the following in the SMMU driver when attaching
> a PCI device:
>
> id = alloc_stream_id(...);
> pci_set_streamid(pci-dev, id);
>
> or
>
> id = pci_get_streamid(pci-dev);
>
> ...i.e the PCI RC could allocate (from some TBD
> allocator) and set the stream ID itself.
>
> Not sure how big a deal it is to extend PCI RC interfaces for
> something like that.
I think both interfaces have their place; the latter for hotpluggable buses
and the former for static allocation (i.e. set once). The only way to see
how well it works is to try implementing it, I guess.
Will