From: Mohan Kumar <[email protected]>
Tegra ADMA HW supports multiple PAGES for virtualization, to
support virtualization
- reg-names property has been added to DT binding for the hypervisor mode.
- In hypervisor mode the ADMA global registers are not accessed by guest.
Mohan Kumar (2):
dt-bindings: dma: Add reg-names to nvidia,tegra210-adma
dmaengine: tegra210-adma: Add support for ADMA virtualization
.../bindings/dma/nvidia,tegra210-adma.yaml | 10 +++++
drivers/dma/tegra210-adma.c | 44 +++++++++++++++----
2 files changed, 46 insertions(+), 8 deletions(-)
--
2.45.1
From: Mohan Kumar <[email protected]>
For Non-Hypervisor mode, Tegra ADMA driver requires the register
resource range to include both global and channel page in the reg
entry. For Hypervisor more, Tegra ADMA driver requires only the
channel page and global page range is not allowed for access.
Add reg-names DT binding for Hypervisor mode to help driver to
differentiate the config between Hypervisor and Non-Hypervisor
mode of execution.
Signed-off-by: Mohan Kumar <[email protected]>
Signed-off-by: Sameer Pujar <[email protected]>
---
.../devicetree/bindings/dma/nvidia,tegra210-adma.yaml | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
index 877147e95ecc..ede47f4a3eec 100644
--- a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
+++ b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
@@ -29,8 +29,18 @@ properties:
- const: nvidia,tegra186-adma
reg:
+ description: |
+ For hypervisor mode, the address range should include a
+ ADMA channel page address range, for non-hypervisor mode
+ it starts with ADMA base address covering Global and Channel
+ page address range.
maxItems: 1
+ reg-names:
+ description: only required for Hypervisor mode.
+ items:
+ - const: vm
+
interrupts:
description: |
Should contain all of the per-channel DMA interrupts in
--
2.45.1
On 21/05/2024 13:08, Sameer Pujar wrote:
> From: Mohan Kumar <[email protected]>
>
> For Non-Hypervisor mode, Tegra ADMA driver requires the register
> resource range to include both global and channel page in the reg
> entry. For Hypervisor more, Tegra ADMA driver requires only the
> channel page and global page range is not allowed for access.
>
> Add reg-names DT binding for Hypervisor mode to help driver to
> differentiate the config between Hypervisor and Non-Hypervisor
> mode of execution.
>
> Signed-off-by: Mohan Kumar <[email protected]>
> Signed-off-by: Sameer Pujar <[email protected]>
> ---
> .../devicetree/bindings/dma/nvidia,tegra210-adma.yaml | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> index 877147e95ecc..ede47f4a3eec 100644
> --- a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> +++ b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> @@ -29,8 +29,18 @@ properties:
> - const: nvidia,tegra186-adma
>
> reg:
> + description: |
> + For hypervisor mode, the address range should include a
> + ADMA channel page address range, for non-hypervisor mode
> + it starts with ADMA base address covering Global and Channel
> + page address range.
> maxItems: 1
>
> + reg-names:
> + description: only required for Hypervisor mode.
This does not work like that. I provide vm entry for non-hypervisor mode
and what? You claim it is virtualized?
Drop property.
Best regards,
Krzysztof
On 21-05-2024 17:23, Krzysztof Kozlowski wrote:
> On 21/05/2024 13:08, Sameer Pujar wrote:
>> From: Mohan Kumar <[email protected]>
>>
>> For Non-Hypervisor mode, Tegra ADMA driver requires the register
>> resource range to include both global and channel page in the reg
>> entry. For Hypervisor more, Tegra ADMA driver requires only the
>> channel page and global page range is not allowed for access.
>>
>> Add reg-names DT binding for Hypervisor mode to help driver to
>> differentiate the config between Hypervisor and Non-Hypervisor
>> mode of execution.
>>
>> Signed-off-by: Mohan Kumar <[email protected]>
>> Signed-off-by: Sameer Pujar <[email protected]>
>> ---
>> .../devicetree/bindings/dma/nvidia,tegra210-adma.yaml | 10 ++++++++++
>> 1 file changed, 10 insertions(+)
>>
>> diff --git a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>> index 877147e95ecc..ede47f4a3eec 100644
>> --- a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>> +++ b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>> @@ -29,8 +29,18 @@ properties:
>> - const: nvidia,tegra186-adma
>>
>> reg:
>> + description: |
>> + For hypervisor mode, the address range should include a
>> + ADMA channel page address range, for non-hypervisor mode
>> + it starts with ADMA base address covering Global and Channel
>> + page address range.
>> maxItems: 1
>>
>> + reg-names:
>> + description: only required for Hypervisor mode.
> This does not work like that. I provide vm entry for non-hypervisor mode
> and what? You claim it is virtualized?
>
> Drop property.
With 'vm' entry added for hypervisor mode, the 'reg' address range needs
to be updated to use channel specific region only. This is used to
inform driver to skip global regions which is taken care by hypervisor.
This is expected to be used in the scenario where Linux acts as a
virtual machine (VM). May be the hypervisor mode gives a different
impression here? Sorry, I did not understand what dropping the property
exactly means here.
On 22/05/2024 07:35, Sameer Pujar wrote:
>
>
> On 21-05-2024 17:23, Krzysztof Kozlowski wrote:
>> On 21/05/2024 13:08, Sameer Pujar wrote:
>>> From: Mohan Kumar <[email protected]>
>>>
>>> For Non-Hypervisor mode, Tegra ADMA driver requires the register
>>> resource range to include both global and channel page in the reg
>>> entry. For Hypervisor more, Tegra ADMA driver requires only the
>>> channel page and global page range is not allowed for access.
>>>
>>> Add reg-names DT binding for Hypervisor mode to help driver to
>>> differentiate the config between Hypervisor and Non-Hypervisor
>>> mode of execution.
>>>
>>> Signed-off-by: Mohan Kumar <[email protected]>
>>> Signed-off-by: Sameer Pujar <[email protected]>
>>> ---
>>> .../devicetree/bindings/dma/nvidia,tegra210-adma.yaml | 10 ++++++++++
>>> 1 file changed, 10 insertions(+)
>>>
>>> diff --git a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>>> index 877147e95ecc..ede47f4a3eec 100644
>>> --- a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>>> +++ b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>>> @@ -29,8 +29,18 @@ properties:
>>> - const: nvidia,tegra186-adma
>>>
>>> reg:
>>> + description: |
>>> + For hypervisor mode, the address range should include a
>>> + ADMA channel page address range, for non-hypervisor mode
>>> + it starts with ADMA base address covering Global and Channel
>>> + page address range.
>>> maxItems: 1
>>>
>>> + reg-names:
>>> + description: only required for Hypervisor mode.
>> This does not work like that. I provide vm entry for non-hypervisor mode
>> and what? You claim it is virtualized?
>>
>> Drop property.
>
> With 'vm' entry added for hypervisor mode, the 'reg' address range needs
> to be updated to use channel specific region only. This is used to
> inform driver to skip global regions which is taken care by hypervisor.
> This is expected to be used in the scenario where Linux acts as a
> virtual machine (VM). May be the hypervisor mode gives a different
> impression here? Sorry, I did not understand what dropping the property
> exactly means here.
It was imperative. Drop it. Remove it. I provided explanation why.
Also, drop unneeded |.
Best regards,
Krzysztof
On 22-05-2024 12:17, Krzysztof Kozlowski wrote:
> On 22/05/2024 07:35, Sameer Pujar wrote:
>> On 21-05-2024 17:23, Krzysztof Kozlowski wrote:
>>> On 21/05/2024 13:08, Sameer Pujar wrote:
>>>> From: Mohan Kumar <[email protected]>
>>>>
>>>> For Non-Hypervisor mode, Tegra ADMA driver requires the register
>>>> resource range to include both global and channel page in the reg
>>>> entry. For Hypervisor more, Tegra ADMA driver requires only the
>>>> channel page and global page range is not allowed for access.
>>>>
>>>> Add reg-names DT binding for Hypervisor mode to help driver to
>>>> differentiate the config between Hypervisor and Non-Hypervisor
>>>> mode of execution.
>>>>
>>>> Signed-off-by: Mohan Kumar <[email protected]>
>>>> Signed-off-by: Sameer Pujar <[email protected]>
>>>> ---
>>>> .../devicetree/bindings/dma/nvidia,tegra210-adma.yaml | 10 ++++++++++
>>>> 1 file changed, 10 insertions(+)
>>>>
>>>> diff --git a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>>>> index 877147e95ecc..ede47f4a3eec 100644
>>>> --- a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>>>> +++ b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>>>> @@ -29,8 +29,18 @@ properties:
>>>> - const: nvidia,tegra186-adma
>>>>
>>>> reg:
>>>> + description: |
>>>> + For hypervisor mode, the address range should include a
>>>> + ADMA channel page address range, for non-hypervisor mode
>>>> + it starts with ADMA base address covering Global and Channel
>>>> + page address range.
>>>> maxItems: 1
>>>>
>>>> + reg-names:
>>>> + description: only required for Hypervisor mode.
>>> This does not work like that. I provide vm entry for non-hypervisor mode
>>> and what? You claim it is virtualized?
>>>
>>> Drop property.
>> With 'vm' entry added for hypervisor mode, the 'reg' address range needs
>> to be updated to use channel specific region only. This is used to
>> inform driver to skip global regions which is taken care by hypervisor.
>> This is expected to be used in the scenario where Linux acts as a
>> virtual machine (VM). May be the hypervisor mode gives a different
>> impression here? Sorry, I did not understand what dropping the property
>> exactly means here.
> It was imperative. Drop it. Remove it. I provided explanation why.
The driver doesn't know if it is operated in a native config or in the
hypervisor config based on the 'reg' address range alone. So 'vm' entry
with restricted 'reg' range is used to differentiate here for the
hypervisor config. Just adding 'vm' entry won't be enough, the 'reg'
region must be updated as well to have expected behavior. Not sure how
this dependency can be enforced in the schema.
> Also, drop unneeded |.
will drop.
On 22/05/2024 09:43, Sameer Pujar wrote:
>
>
> On 22-05-2024 12:17, Krzysztof Kozlowski wrote:
>> On 22/05/2024 07:35, Sameer Pujar wrote:
>>> On 21-05-2024 17:23, Krzysztof Kozlowski wrote:
>>>> On 21/05/2024 13:08, Sameer Pujar wrote:
>>>>> From: Mohan Kumar <[email protected]>
>>>>>
>>>>> For Non-Hypervisor mode, Tegra ADMA driver requires the register
>>>>> resource range to include both global and channel page in the reg
>>>>> entry. For Hypervisor more, Tegra ADMA driver requires only the
>>>>> channel page and global page range is not allowed for access.
>>>>>
>>>>> Add reg-names DT binding for Hypervisor mode to help driver to
>>>>> differentiate the config between Hypervisor and Non-Hypervisor
>>>>> mode of execution.
>>>>>
>>>>> Signed-off-by: Mohan Kumar <[email protected]>
>>>>> Signed-off-by: Sameer Pujar <[email protected]>
>>>>> ---
>>>>> .../devicetree/bindings/dma/nvidia,tegra210-adma.yaml | 10 ++++++++++
>>>>> 1 file changed, 10 insertions(+)
>>>>>
>>>>> diff --git a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>>>>> index 877147e95ecc..ede47f4a3eec 100644
>>>>> --- a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>>>>> +++ b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>>>>> @@ -29,8 +29,18 @@ properties:
>>>>> - const: nvidia,tegra186-adma
>>>>>
>>>>> reg:
>>>>> + description: |
>>>>> + For hypervisor mode, the address range should include a
>>>>> + ADMA channel page address range, for non-hypervisor mode
>>>>> + it starts with ADMA base address covering Global and Channel
>>>>> + page address range.
>>>>> maxItems: 1
>>>>>
>>>>> + reg-names:
>>>>> + description: only required for Hypervisor mode.
>>>> This does not work like that. I provide vm entry for non-hypervisor mode
>>>> and what? You claim it is virtualized?
>>>>
>>>> Drop property.
>>> With 'vm' entry added for hypervisor mode, the 'reg' address range needs
>>> to be updated to use channel specific region only. This is used to
>>> inform driver to skip global regions which is taken care by hypervisor.
>>> This is expected to be used in the scenario where Linux acts as a
>>> virtual machine (VM). May be the hypervisor mode gives a different
>>> impression here? Sorry, I did not understand what dropping the property
>>> exactly means here.
>> It was imperative. Drop it. Remove it. I provided explanation why.
>
> The driver doesn't know if it is operated in a native config or in the
> hypervisor config based on the 'reg' address range alone. So 'vm' entry
> with restricted 'reg' range is used to differentiate here for the
> hypervisor config. Just adding 'vm' entry won't be enough, the 'reg'
> region must be updated as well to have expected behavior. Not sure how
> this dependency can be enforced in the schema.
That's not a unusual problem, so please come with a solution for your
entire subarch. We've been discussing similar topic in terms of SCMI
controlled resources (see talk on Linaro Connect a week ago:
https://www.kitefor.events/events/linaro-connect-24/submissions/161 I
don't know where is recording or slides, see also discussions on mailing
lists about it), which is not that far away from the problem here. Other
platforms and maybe nvidia had as well changes in IO space for
virtualized configuration.
Come with unified approach FOR ALL your devices, not only this one
(that's kind of basic thing we keep repeating... don't solve only one
your problem), do not abuse the regular property, because as I said:
reg-names will be provided as well in non-vm case and then your entire
logic is wrong. The purpose of reg-names is not to tell whether you have
or have not virtualized environment.
Best regards,
Krzysztof
On Wed May 22, 2024 at 1:29 PM CEST, Krzysztof Kozlowski wrote:
> On 22/05/2024 09:43, Sameer Pujar wrote:
> >
> >
> > On 22-05-2024 12:17, Krzysztof Kozlowski wrote:
> >> On 22/05/2024 07:35, Sameer Pujar wrote:
> >>> On 21-05-2024 17:23, Krzysztof Kozlowski wrote:
> >>>> On 21/05/2024 13:08, Sameer Pujar wrote:
> >>>>> From: Mohan Kumar <[email protected]>
> >>>>>
> >>>>> For Non-Hypervisor mode, Tegra ADMA driver requires the register
> >>>>> resource range to include both global and channel page in the reg
> >>>>> entry. For Hypervisor more, Tegra ADMA driver requires only the
> >>>>> channel page and global page range is not allowed for access.
> >>>>>
> >>>>> Add reg-names DT binding for Hypervisor mode to help driver to
> >>>>> differentiate the config between Hypervisor and Non-Hypervisor
> >>>>> mode of execution.
> >>>>>
> >>>>> Signed-off-by: Mohan Kumar <[email protected]>
> >>>>> Signed-off-by: Sameer Pujar <[email protected]>
> >>>>> ---
> >>>>> .../devicetree/bindings/dma/nvidia,tegra210-adma.yaml | 10 ++++++++++
> >>>>> 1 file changed, 10 insertions(+)
> >>>>>
> >>>>> diff --git a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> >>>>> index 877147e95ecc..ede47f4a3eec 100644
> >>>>> --- a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> >>>>> +++ b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> >>>>> @@ -29,8 +29,18 @@ properties:
> >>>>> - const: nvidia,tegra186-adma
> >>>>>
> >>>>> reg:
> >>>>> + description: |
> >>>>> + For hypervisor mode, the address range should include a
> >>>>> + ADMA channel page address range, for non-hypervisor mode
> >>>>> + it starts with ADMA base address covering Global and Channel
> >>>>> + page address range.
> >>>>> maxItems: 1
> >>>>>
> >>>>> + reg-names:
> >>>>> + description: only required for Hypervisor mode.
> >>>> This does not work like that. I provide vm entry for non-hypervisor mode
> >>>> and what? You claim it is virtualized?
> >>>>
> >>>> Drop property.
> >>> With 'vm' entry added for hypervisor mode, the 'reg' address range needs
> >>> to be updated to use channel specific region only. This is used to
> >>> inform driver to skip global regions which is taken care by hypervisor.
> >>> This is expected to be used in the scenario where Linux acts as a
> >>> virtual machine (VM). May be the hypervisor mode gives a different
> >>> impression here? Sorry, I did not understand what dropping the property
> >>> exactly means here.
> >> It was imperative. Drop it. Remove it. I provided explanation why.
> >
> > The driver doesn't know if it is operated in a native config or in the
> > hypervisor config based on the 'reg' address range alone. So 'vm' entry
> > with restricted 'reg' range is used to differentiate here for the
> > hypervisor config. Just adding 'vm' entry won't be enough, the 'reg'
> > region must be updated as well to have expected behavior. Not sure how
> > this dependency can be enforced in the schema.
>
> That's not a unusual problem, so please come with a solution for your
> entire subarch. We've been discussing similar topic in terms of SCMI
> controlled resources (see talk on Linaro Connect a week ago:
> https://www.kitefor.events/events/linaro-connect-24/submissions/161 I
> don't know where is recording or slides, see also discussions on mailing
> lists about it), which is not that far away from the problem here. Other
> platforms and maybe nvidia had as well changes in IO space for
> virtualized configuration.
>
> Come with unified approach FOR ALL your devices, not only this one
> (that's kind of basic thing we keep repeating... don't solve only one
> your problem), do not abuse the regular property, because as I said:
> reg-names will be provided as well in non-vm case and then your entire
> logic is wrong. The purpose of reg-names is not to tell whether you have
> or have not virtualized environment.
This isn't strictly about telling whether this is a virtualized
environment or not. Unfortunately the bindings don't make that very
clear, so let me try to give a bit more background.
On Tegra devices the register regions associated with a device are
usually split up into 64 KiB chunks.
One of these chunks, usually the first one, is a global region that
contains registers that configure the device as a whole. This is usually
privileged and accessible only to the hypervisor.
Subsequent regions are meant to be assigned to individual VMs. Often the
regions take the form of "channels", so they are instances of the same
register block and control that separate slice of the hardware.
What makes this a bit confusing is that for the sake of simplicity (and,
I guess, lack of foresight) the original bindings were written in a way
to encompass all registers without making that distinction. This worked
fine because we've only ever run Linux as host OS where it has access to
all those registers.
However, when we move to virtualized environments that no longer works.
Given the above, we can't read any registers in order to probe whether
we run as a guest or not. Trying to access any of the global registers
from a VM simply won't work and may crash the system. None of the
"channel" registers contain information indicating host vs. guest
either.
In order to make this work we need to more fine-grainedly specify the
register layout. I think the binding changes here aren't sufficient to
do that, though.
Currently we have this for the ADMA controller:
dma-controller@2930000 {
reg = <0x0 0x02930000 0x0 0x20000>;
};
This contains the global registers (0x2930000-0x293ffff) and the first
page/channel registers (0x2940000-0x294ffff) in one "reg" entry. Instead
I think what we need is this:
dma-controller@2930000 {
reg = <0x0 0x02930000 0x0 0x10000>,
<0x0 0x02940000 0x0 0x10000>,
<0x0 0x02950000 0x0 0x10000>,
<0x0 0x02960000 0x0 0x10000>,
<0x0 0x02970000 0x0 0x10000>;
reg-names = "global", "page0", "page1", "page2",
"page3";
};
That describes the device fully, but each of these entries is optional.
If "global" is present it means we are a hypervisor (or host OS). If an
additional "page" entry is present, we can also use those resources to
stream audio data.
If "global" is not present, we know we are not a hypervisor and those
registers cannot be accessed. This would be the typical case for a guest
OS which has access only to the listed "page" entries.
For backwards-compatibility with the existing bindings we should be able
to fallback to the singular register region and partition it up in the
driver as necessary.
This is an approach that we've already implemented for certain devices
such as host1x and Ethernet where a similar split exists. I suspect that
we'll need to do this kind of split in a number of other bindings as
well.
Thierry
On 24/05/2024 09:36, Thierry Reding wrote:
> On Wed May 22, 2024 at 1:29 PM CEST, Krzysztof Kozlowski wrote:
>> On 22/05/2024 09:43, Sameer Pujar wrote:
>>>
>>>
>>> On 22-05-2024 12:17, Krzysztof Kozlowski wrote:
>>>> On 22/05/2024 07:35, Sameer Pujar wrote:
>>>>> On 21-05-2024 17:23, Krzysztof Kozlowski wrote:
>>>>>> On 21/05/2024 13:08, Sameer Pujar wrote:
>>>>>>> From: Mohan Kumar <[email protected]>
>>>>>>>
>>>>>>> For Non-Hypervisor mode, Tegra ADMA driver requires the register
>>>>>>> resource range to include both global and channel page in the reg
>>>>>>> entry. For Hypervisor more, Tegra ADMA driver requires only the
>>>>>>> channel page and global page range is not allowed for access.
>>>>>>>
>>>>>>> Add reg-names DT binding for Hypervisor mode to help driver to
>>>>>>> differentiate the config between Hypervisor and Non-Hypervisor
>>>>>>> mode of execution.
>>>>>>>
>>>>>>> Signed-off-by: Mohan Kumar <[email protected]>
>>>>>>> Signed-off-by: Sameer Pujar <[email protected]>
>>>>>>> ---
>>>>>>> .../devicetree/bindings/dma/nvidia,tegra210-adma.yaml | 10 ++++++++++
>>>>>>> 1 file changed, 10 insertions(+)
>>>>>>>
>>>>>>> diff --git a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>>>>>>> index 877147e95ecc..ede47f4a3eec 100644
>>>>>>> --- a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>>>>>>> +++ b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>>>>>>> @@ -29,8 +29,18 @@ properties:
>>>>>>> - const: nvidia,tegra186-adma
>>>>>>>
>>>>>>> reg:
>>>>>>> + description: |
>>>>>>> + For hypervisor mode, the address range should include a
>>>>>>> + ADMA channel page address range, for non-hypervisor mode
>>>>>>> + it starts with ADMA base address covering Global and Channel
>>>>>>> + page address range.
>>>>>>> maxItems: 1
>>>>>>>
>>>>>>> + reg-names:
>>>>>>> + description: only required for Hypervisor mode.
>>>>>> This does not work like that. I provide vm entry for non-hypervisor mode
>>>>>> and what? You claim it is virtualized?
>>>>>>
>>>>>> Drop property.
>>>>> With 'vm' entry added for hypervisor mode, the 'reg' address range needs
>>>>> to be updated to use channel specific region only. This is used to
>>>>> inform driver to skip global regions which is taken care by hypervisor.
>>>>> This is expected to be used in the scenario where Linux acts as a
>>>>> virtual machine (VM). May be the hypervisor mode gives a different
>>>>> impression here? Sorry, I did not understand what dropping the property
>>>>> exactly means here.
>>>> It was imperative. Drop it. Remove it. I provided explanation why.
>>>
>>> The driver doesn't know if it is operated in a native config or in the
>>> hypervisor config based on the 'reg' address range alone. So 'vm' entry
>>> with restricted 'reg' range is used to differentiate here for the
>>> hypervisor config. Just adding 'vm' entry won't be enough, the 'reg'
>>> region must be updated as well to have expected behavior. Not sure how
>>> this dependency can be enforced in the schema.
>>
>> That's not a unusual problem, so please come with a solution for your
>> entire subarch. We've been discussing similar topic in terms of SCMI
>> controlled resources (see talk on Linaro Connect a week ago:
>> https://www.kitefor.events/events/linaro-connect-24/submissions/161 I
>> don't know where is recording or slides, see also discussions on mailing
>> lists about it), which is not that far away from the problem here. Other
>> platforms and maybe nvidia had as well changes in IO space for
>> virtualized configuration.
>>
>> Come with unified approach FOR ALL your devices, not only this one
>> (that's kind of basic thing we keep repeating... don't solve only one
>> your problem), do not abuse the regular property, because as I said:
>> reg-names will be provided as well in non-vm case and then your entire
>> logic is wrong. The purpose of reg-names is not to tell whether you have
>> or have not virtualized environment.
>
> This isn't strictly about telling whether this is a virtualized
> environment or not. Unfortunately the bindings don't make that very
> clear, so let me try to give a bit more background.
>
> On Tegra devices the register regions associated with a device are
> usually split up into 64 KiB chunks.
So describing it as one IO region was incorrect from the start and you
want to fix it by adding one more incorrect description: making first
item meaning two different things. Sorry, that's not a correct way to
fix things.
Items are defined, thus first item is always expected to be what the
binding already said. Adding reg-names changes nothing, because (as
repeated many times) xxx-names is just a helper. Items are already defined.
>
> One of these chunks, usually the first one, is a global region that
> contains registers that configure the device as a whole. This is usually
> privileged and accessible only to the hypervisor.
>
> Subsequent regions are meant to be assigned to individual VMs. Often the
> regions take the form of "channels", so they are instances of the same
> register block and control that separate slice of the hardware.
>
> What makes this a bit confusing is that for the sake of simplicity (and,
> I guess, lack of foresight) the original bindings were written in a way
> to encompass all registers without making that distinction. This worked
> fine because we've only ever run Linux as host OS where it has access to
> all those registers.
>
> However, when we move to virtualized environments that no longer works.
>
> Given the above, we can't read any registers in order to probe whether
> we run as a guest or not. Trying to access any of the global registers
> from a VM simply won't work and may crash the system. None of the
> "channel" registers contain information indicating host vs. guest
> either.
I don't understand how it differs from what I said - you want to
indicate that you run in virtualized environment and not all resources
are accessible.
The device still has the first (global) address, just it is not
available due to hypervisor.
>
> In order to make this work we need to more fine-grainedly specify the
> register layout. I think the binding changes here aren't sufficient to
> do that, though.
>
> Currently we have this for the ADMA controller:
>
> dma-controller@2930000 {
> reg = <0x0 0x02930000 0x0 0x20000>;
> };
>
> This contains the global registers (0x2930000-0x293ffff) and the first
> page/channel registers (0x2940000-0x294ffff) in one "reg" entry. Instead
> I think what we need is this:
>
> dma-controller@2930000 {
> reg = <0x0 0x02930000 0x0 0x10000>,
> <0x0 0x02940000 0x0 0x10000>,
> <0x0 0x02950000 0x0 0x10000>,
> <0x0 0x02960000 0x0 0x10000>,
> <0x0 0x02970000 0x0 0x10000>;
> reg-names = "global", "page0", "page1", "page2",
> "page3";
> };
>
> That describes the device fully, but each of these entries is optional.
> If "global" is present it means we are a hypervisor (or host OS). If an
> additional "page" entry is present, we can also use those resources to
> stream audio data.
>
> If "global" is not present, we know we are not a hypervisor and those
> registers cannot be accessed. This would be the typical case for a guest
> OS which has access only to the listed "page" entries.
>
> For backwards-compatibility with the existing bindings we should be able
> to fallback to the singular register region and partition it up in the
> driver as necessary.
>
> This is an approach that we've already implemented for certain devices
> such as host1x and Ethernet where a similar split exists. I suspect that
> we'll need to do this kind of split in a number of other bindings as
> well.
>
Best regards,
Krzysztof
On Fri, May 24, 2024 at 09:36:08AM +0200, Thierry Reding wrote:
> On Wed May 22, 2024 at 1:29 PM CEST, Krzysztof Kozlowski wrote:
> > On 22/05/2024 09:43, Sameer Pujar wrote:
> > >
> > >
> > > On 22-05-2024 12:17, Krzysztof Kozlowski wrote:
> > >> On 22/05/2024 07:35, Sameer Pujar wrote:
> > >>> On 21-05-2024 17:23, Krzysztof Kozlowski wrote:
> > >>>> On 21/05/2024 13:08, Sameer Pujar wrote:
> > >>>>> From: Mohan Kumar <[email protected]>
> > >>>>>
> > >>>>> For Non-Hypervisor mode, Tegra ADMA driver requires the register
> > >>>>> resource range to include both global and channel page in the reg
> > >>>>> entry. For Hypervisor more, Tegra ADMA driver requires only the
> > >>>>> channel page and global page range is not allowed for access.
> > >>>>>
> > >>>>> Add reg-names DT binding for Hypervisor mode to help driver to
> > >>>>> differentiate the config between Hypervisor and Non-Hypervisor
> > >>>>> mode of execution.
> > >>>>>
> > >>>>> Signed-off-by: Mohan Kumar <[email protected]>
> > >>>>> Signed-off-by: Sameer Pujar <[email protected]>
> > >>>>> ---
> > >>>>> .../devicetree/bindings/dma/nvidia,tegra210-adma.yaml | 10 ++++++++++
> > >>>>> 1 file changed, 10 insertions(+)
> > >>>>>
> > >>>>> diff --git a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> > >>>>> index 877147e95ecc..ede47f4a3eec 100644
> > >>>>> --- a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> > >>>>> +++ b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> > >>>>> @@ -29,8 +29,18 @@ properties:
> > >>>>> - const: nvidia,tegra186-adma
> > >>>>>
> > >>>>> reg:
> > >>>>> + description: |
> > >>>>> + For hypervisor mode, the address range should include a
> > >>>>> + ADMA channel page address range, for non-hypervisor mode
> > >>>>> + it starts with ADMA base address covering Global and Channel
> > >>>>> + page address range.
> > >>>>> maxItems: 1
> > >>>>>
> > >>>>> + reg-names:
> > >>>>> + description: only required for Hypervisor mode.
> > >>>> This does not work like that. I provide vm entry for non-hypervisor mode
> > >>>> and what? You claim it is virtualized?
> > >>>>
> > >>>> Drop property.
> > >>> With 'vm' entry added for hypervisor mode, the 'reg' address range needs
> > >>> to be updated to use channel specific region only. This is used to
> > >>> inform driver to skip global regions which is taken care by hypervisor.
> > >>> This is expected to be used in the scenario where Linux acts as a
> > >>> virtual machine (VM). May be the hypervisor mode gives a different
> > >>> impression here? Sorry, I did not understand what dropping the property
> > >>> exactly means here.
> > >> It was imperative. Drop it. Remove it. I provided explanation why.
> > >
> > > The driver doesn't know if it is operated in a native config or in the
> > > hypervisor config based on the 'reg' address range alone. So 'vm' entry
> > > with restricted 'reg' range is used to differentiate here for the
> > > hypervisor config. Just adding 'vm' entry won't be enough, the 'reg'
> > > region must be updated as well to have expected behavior. Not sure how
> > > this dependency can be enforced in the schema.
> >
> > That's not a unusual problem, so please come with a solution for your
> > entire subarch. We've been discussing similar topic in terms of SCMI
> > controlled resources (see talk on Linaro Connect a week ago:
> > https://www.kitefor.events/events/linaro-connect-24/submissions/161 I
> > don't know where is recording or slides, see also discussions on mailing
> > lists about it), which is not that far away from the problem here. Other
> > platforms and maybe nvidia had as well changes in IO space for
> > virtualized configuration.
> >
> > Come with unified approach FOR ALL your devices, not only this one
> > (that's kind of basic thing we keep repeating... don't solve only one
> > your problem), do not abuse the regular property, because as I said:
> > reg-names will be provided as well in non-vm case and then your entire
> > logic is wrong. The purpose of reg-names is not to tell whether you have
> > or have not virtualized environment.
>
> This isn't strictly about telling whether this is a virtualized
> environment or not. Unfortunately the bindings don't make that very
> clear, so let me try to give a bit more background.
>
> On Tegra devices the register regions associated with a device are
> usually split up into 64 KiB chunks.
>
> One of these chunks, usually the first one, is a global region that
> contains registers that configure the device as a whole. This is usually
> privileged and accessible only to the hypervisor.
>
> Subsequent regions are meant to be assigned to individual VMs. Often the
> regions take the form of "channels", so they are instances of the same
> register block and control that separate slice of the hardware.
>
> What makes this a bit confusing is that for the sake of simplicity (and,
> I guess, lack of foresight) the original bindings were written in a way
> to encompass all registers without making that distinction. This worked
> fine because we've only ever run Linux as host OS where it has access to
> all those registers.
>
> However, when we move to virtualized environments that no longer works.
>
> Given the above, we can't read any registers in order to probe whether
> we run as a guest or not. Trying to access any of the global registers
> from a VM simply won't work and may crash the system. None of the
> "channel" registers contain information indicating host vs. guest
> either.
>
> In order to make this work we need to more fine-grainedly specify the
> register layout. I think the binding changes here aren't sufficient to
> do that, though.
>
> Currently we have this for the ADMA controller:
>
> dma-controller@2930000 {
> reg = <0x0 0x02930000 0x0 0x20000>;
> };
>
> This contains the global registers (0x2930000-0x293ffff) and the first
> page/channel registers (0x2940000-0x294ffff) in one "reg" entry. Instead
> I think what we need is this:
>
> dma-controller@2930000 {
> reg = <0x0 0x02930000 0x0 0x10000>,
> <0x0 0x02940000 0x0 0x10000>,
> <0x0 0x02950000 0x0 0x10000>,
> <0x0 0x02960000 0x0 0x10000>,
> <0x0 0x02970000 0x0 0x10000>;
> reg-names = "global", "page0", "page1", "page2",
> "page3";
> };
>
> That describes the device fully, but each of these entries is optional.
> If "global" is present it means we are a hypervisor (or host OS). If an
> additional "page" entry is present, we can also use those resources to
> stream audio data.
>
> If "global" is not present, we know we are not a hypervisor and those
> registers cannot be accessed. This would be the typical case for a guest
> OS which has access only to the listed "page" entries.
>
> For backwards-compatibility with the existing bindings we should be able
> to fallback to the singular register region and partition it up in the
> driver as necessary.
>
> This is an approach that we've already implemented for certain devices
> such as host1x and Ethernet where a similar split exists. I suspect that
> we'll need to do this kind of split in a number of other bindings as
> well.
In a VM is a different (being a subset) programming model, so why not
just a new compatible for virtualized case. That's what we'd do if
actual h/w registers changed from one device to the next.
Rob
On Tue May 28, 2024 at 5:35 PM CEST, Rob Herring wrote:
> On Fri, May 24, 2024 at 09:36:08AM +0200, Thierry Reding wrote:
> > On Wed May 22, 2024 at 1:29 PM CEST, Krzysztof Kozlowski wrote:
> > > On 22/05/2024 09:43, Sameer Pujar wrote:
> > > >
> > > >
> > > > On 22-05-2024 12:17, Krzysztof Kozlowski wrote:
> > > >> On 22/05/2024 07:35, Sameer Pujar wrote:
> > > >>> On 21-05-2024 17:23, Krzysztof Kozlowski wrote:
> > > >>>> On 21/05/2024 13:08, Sameer Pujar wrote:
> > > >>>>> From: Mohan Kumar <[email protected]>
> > > >>>>>
> > > >>>>> For Non-Hypervisor mode, Tegra ADMA driver requires the register
> > > >>>>> resource range to include both global and channel page in the reg
> > > >>>>> entry. For Hypervisor more, Tegra ADMA driver requires only the
> > > >>>>> channel page and global page range is not allowed for access.
> > > >>>>>
> > > >>>>> Add reg-names DT binding for Hypervisor mode to help driver to
> > > >>>>> differentiate the config between Hypervisor and Non-Hypervisor
> > > >>>>> mode of execution.
> > > >>>>>
> > > >>>>> Signed-off-by: Mohan Kumar <[email protected]>
> > > >>>>> Signed-off-by: Sameer Pujar <[email protected]>
> > > >>>>> ---
> > > >>>>> .../devicetree/bindings/dma/nvidia,tegra210-adma.yaml | 10 ++++++++++
> > > >>>>> 1 file changed, 10 insertions(+)
> > > >>>>>
> > > >>>>> diff --git a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> > > >>>>> index 877147e95ecc..ede47f4a3eec 100644
> > > >>>>> --- a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> > > >>>>> +++ b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> > > >>>>> @@ -29,8 +29,18 @@ properties:
> > > >>>>> - const: nvidia,tegra186-adma
> > > >>>>>
> > > >>>>> reg:
> > > >>>>> + description: |
> > > >>>>> + For hypervisor mode, the address range should include a
> > > >>>>> + ADMA channel page address range, for non-hypervisor mode
> > > >>>>> + it starts with ADMA base address covering Global and Channel
> > > >>>>> + page address range.
> > > >>>>> maxItems: 1
> > > >>>>>
> > > >>>>> + reg-names:
> > > >>>>> + description: only required for Hypervisor mode.
> > > >>>> This does not work like that. I provide vm entry for non-hypervisor mode
> > > >>>> and what? You claim it is virtualized?
> > > >>>>
> > > >>>> Drop property.
> > > >>> With 'vm' entry added for hypervisor mode, the 'reg' address range needs
> > > >>> to be updated to use channel specific region only. This is used to
> > > >>> inform driver to skip global regions which is taken care by hypervisor.
> > > >>> This is expected to be used in the scenario where Linux acts as a
> > > >>> virtual machine (VM). May be the hypervisor mode gives a different
> > > >>> impression here? Sorry, I did not understand what dropping the property
> > > >>> exactly means here.
> > > >> It was imperative. Drop it. Remove it. I provided explanation why.
> > > >
> > > > The driver doesn't know if it is operated in a native config or in the
> > > > hypervisor config based on the 'reg' address range alone. So 'vm' entry
> > > > with restricted 'reg' range is used to differentiate here for the
> > > > hypervisor config. Just adding 'vm' entry won't be enough, the 'reg'
> > > > region must be updated as well to have expected behavior. Not sure how
> > > > this dependency can be enforced in the schema.
> > >
> > > That's not a unusual problem, so please come with a solution for your
> > > entire subarch. We've been discussing similar topic in terms of SCMI
> > > controlled resources (see talk on Linaro Connect a week ago:
> > > https://www.kitefor.events/events/linaro-connect-24/submissions/161 I
> > > don't know where is recording or slides, see also discussions on mailing
> > > lists about it), which is not that far away from the problem here. Other
> > > platforms and maybe nvidia had as well changes in IO space for
> > > virtualized configuration.
> > >
> > > Come with unified approach FOR ALL your devices, not only this one
> > > (that's kind of basic thing we keep repeating... don't solve only one
> > > your problem), do not abuse the regular property, because as I said:
> > > reg-names will be provided as well in non-vm case and then your entire
> > > logic is wrong. The purpose of reg-names is not to tell whether you have
> > > or have not virtualized environment.
> >
> > This isn't strictly about telling whether this is a virtualized
> > environment or not. Unfortunately the bindings don't make that very
> > clear, so let me try to give a bit more background.
> >
> > On Tegra devices the register regions associated with a device are
> > usually split up into 64 KiB chunks.
> >
> > One of these chunks, usually the first one, is a global region that
> > contains registers that configure the device as a whole. This is usually
> > privileged and accessible only to the hypervisor.
> >
> > Subsequent regions are meant to be assigned to individual VMs. Often the
> > regions take the form of "channels", so they are instances of the same
> > register block and control that separate slice of the hardware.
> >
> > What makes this a bit confusing is that for the sake of simplicity (and,
> > I guess, lack of foresight) the original bindings were written in a way
> > to encompass all registers without making that distinction. This worked
> > fine because we've only ever run Linux as host OS where it has access to
> > all those registers.
> >
> > However, when we move to virtualized environments that no longer works.
> >
> > Given the above, we can't read any registers in order to probe whether
> > we run as a guest or not. Trying to access any of the global registers
> > from a VM simply won't work and may crash the system. None of the
> > "channel" registers contain information indicating host vs. guest
> > either.
> >
> > In order to make this work we need to more fine-grainedly specify the
> > register layout. I think the binding changes here aren't sufficient to
> > do that, though.
> >
> > Currently we have this for the ADMA controller:
> >
> > dma-controller@2930000 {
> > reg = <0x0 0x02930000 0x0 0x20000>;
> > };
> >
> > This contains the global registers (0x2930000-0x293ffff) and the first
> > page/channel registers (0x2940000-0x294ffff) in one "reg" entry. Instead
> > I think what we need is this:
> >
> > dma-controller@2930000 {
> > reg = <0x0 0x02930000 0x0 0x10000>,
> > <0x0 0x02940000 0x0 0x10000>,
> > <0x0 0x02950000 0x0 0x10000>,
> > <0x0 0x02960000 0x0 0x10000>,
> > <0x0 0x02970000 0x0 0x10000>;
> > reg-names = "global", "page0", "page1", "page2",
> > "page3";
> > };
> >
> > That describes the device fully, but each of these entries is optional.
> > If "global" is present it means we are a hypervisor (or host OS). If an
> > additional "page" entry is present, we can also use those resources to
> > stream audio data.
> >
> > If "global" is not present, we know we are not a hypervisor and those
> > registers cannot be accessed. This would be the typical case for a guest
> > OS which has access only to the listed "page" entries.
> >
> > For backwards-compatibility with the existing bindings we should be able
> > to fallback to the singular register region and partition it up in the
> > driver as necessary.
> >
> > This is an approach that we've already implemented for certain devices
> > such as host1x and Ethernet where a similar split exists. I suspect that
> > we'll need to do this kind of split in a number of other bindings as
> > well.
>
> In a VM is a different (being a subset) programming model, so why not
> just a new compatible for virtualized case. That's what we'd do if
> actual h/w registers changed from one device to the next.
I suppose you could argue that way. However, the devices are identical
whether we use them in host or guest mode. The only difference is which
registers we can access. And obviously that in the case where we can
access the "global" registers that we also will access them.
But I don't see that as being a different programming model. We've got a
bunch of parameterization elsewhere in the kernel where we don't resort
to new compatible strings. If you really wanted to you could argue that
adding an interrupt GPIO to a device causes the programming model to be
different from a case where you would otherwise do polling. But we don't
and instead make the GPIO optional so that it can be used if available
and we fall back to polling otherwise.
That's very similar to what we want to do here.
There's also the complication that we'd technically need a third
compatible string for the hypervisor. So instead of one compatible
string paired with reg/reg-names to cover all these cases, we'd end up
with three compatible strings just so we can stick with the single reg
entry. And that's not counting any use-cases we don't know of yet.
So what we really need to resolve here is different use-cases of the
same hardware. A compatible string doesn't seem like the right option
for that. Parameterization is a much better solution to that problem.
Thierry
On Tue May 28, 2024 at 8:48 AM CEST, Krzysztof Kozlowski wrote:
> On 24/05/2024 09:36, Thierry Reding wrote:
> > On Wed May 22, 2024 at 1:29 PM CEST, Krzysztof Kozlowski wrote:
> >> On 22/05/2024 09:43, Sameer Pujar wrote:
> >>>
> >>>
> >>> On 22-05-2024 12:17, Krzysztof Kozlowski wrote:
> >>>> On 22/05/2024 07:35, Sameer Pujar wrote:
> >>>>> On 21-05-2024 17:23, Krzysztof Kozlowski wrote:
> >>>>>> On 21/05/2024 13:08, Sameer Pujar wrote:
> >>>>>>> From: Mohan Kumar <[email protected]>
> >>>>>>>
> >>>>>>> For Non-Hypervisor mode, Tegra ADMA driver requires the register
> >>>>>>> resource range to include both global and channel page in the reg
> >>>>>>> entry. For Hypervisor more, Tegra ADMA driver requires only the
> >>>>>>> channel page and global page range is not allowed for access.
> >>>>>>>
> >>>>>>> Add reg-names DT binding for Hypervisor mode to help driver to
> >>>>>>> differentiate the config between Hypervisor and Non-Hypervisor
> >>>>>>> mode of execution.
> >>>>>>>
> >>>>>>> Signed-off-by: Mohan Kumar <[email protected]>
> >>>>>>> Signed-off-by: Sameer Pujar <[email protected]>
> >>>>>>> ---
> >>>>>>> .../devicetree/bindings/dma/nvidia,tegra210-adma.yaml | 10 ++++++++++
> >>>>>>> 1 file changed, 10 insertions(+)
> >>>>>>>
> >>>>>>> diff --git a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> >>>>>>> index 877147e95ecc..ede47f4a3eec 100644
> >>>>>>> --- a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> >>>>>>> +++ b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> >>>>>>> @@ -29,8 +29,18 @@ properties:
> >>>>>>> - const: nvidia,tegra186-adma
> >>>>>>>
> >>>>>>> reg:
> >>>>>>> + description: |
> >>>>>>> + For hypervisor mode, the address range should include a
> >>>>>>> + ADMA channel page address range, for non-hypervisor mode
> >>>>>>> + it starts with ADMA base address covering Global and Channel
> >>>>>>> + page address range.
> >>>>>>> maxItems: 1
> >>>>>>>
> >>>>>>> + reg-names:
> >>>>>>> + description: only required for Hypervisor mode.
> >>>>>> This does not work like that. I provide vm entry for non-hypervisor mode
> >>>>>> and what? You claim it is virtualized?
> >>>>>>
> >>>>>> Drop property.
> >>>>> With 'vm' entry added for hypervisor mode, the 'reg' address range needs
> >>>>> to be updated to use channel specific region only. This is used to
> >>>>> inform driver to skip global regions which is taken care by hypervisor.
> >>>>> This is expected to be used in the scenario where Linux acts as a
> >>>>> virtual machine (VM). May be the hypervisor mode gives a different
> >>>>> impression here? Sorry, I did not understand what dropping the property
> >>>>> exactly means here.
> >>>> It was imperative. Drop it. Remove it. I provided explanation why.
> >>>
> >>> The driver doesn't know if it is operated in a native config or in the
> >>> hypervisor config based on the 'reg' address range alone. So 'vm' entry
> >>> with restricted 'reg' range is used to differentiate here for the
> >>> hypervisor config. Just adding 'vm' entry won't be enough, the 'reg'
> >>> region must be updated as well to have expected behavior. Not sure how
> >>> this dependency can be enforced in the schema.
> >>
> >> That's not a unusual problem, so please come with a solution for your
> >> entire subarch. We've been discussing similar topic in terms of SCMI
> >> controlled resources (see talk on Linaro Connect a week ago:
> >> https://www.kitefor.events/events/linaro-connect-24/submissions/161 I
> >> don't know where is recording or slides, see also discussions on mailing
> >> lists about it), which is not that far away from the problem here. Other
> >> platforms and maybe nvidia had as well changes in IO space for
> >> virtualized configuration.
> >>
> >> Come with unified approach FOR ALL your devices, not only this one
> >> (that's kind of basic thing we keep repeating... don't solve only one
> >> your problem), do not abuse the regular property, because as I said:
> >> reg-names will be provided as well in non-vm case and then your entire
> >> logic is wrong. The purpose of reg-names is not to tell whether you have
> >> or have not virtualized environment.
> >
> > This isn't strictly about telling whether this is a virtualized
> > environment or not. Unfortunately the bindings don't make that very
> > clear, so let me try to give a bit more background.
> >
> > On Tegra devices the register regions associated with a device are
> > usually split up into 64 KiB chunks.
>
> So describing it as one IO region was incorrect from the start and you
> want to fix it by adding one more incorrect description: making first
> item meaning two different things. Sorry, that's not a correct way to
> fix things.
Yes, describing this as one I/O region was incorrect, and in hindsight
it should have been done differently.
However, I don't think it's correct to describe this as adding one more
incorrect description. Instead, what this does is add reg-names to
provide additional context so that the operating system can make the
necessary decisions as to what is allowed and what isn't.
In the absence of a reg-names property the current definition of the DT
bindings applies, so it means the region represents the entirety of the
device's I/O register space. That's one particular use-case for this
device.
For additional use-cases we can then use reg-names to differentiate
between what separate regions are and use them accordingly.
> Items are defined, thus first item is always expected to be what the
> binding already said. Adding reg-names changes nothing, because (as
> repeated many times) xxx-names is just a helper. Items are already defined.
I don't understand what you're trying to say here. I suppose adding
reg-names alone indeed doesn't change anything. But the point is that
once added we can now use these properties, at which point of course
things change.
> > One of these chunks, usually the first one, is a global region that
> > contains registers that configure the device as a whole. This is usually
> > privileged and accessible only to the hypervisor.
> >
> > Subsequent regions are meant to be assigned to individual VMs. Often the
> > regions take the form of "channels", so they are instances of the same
> > register block and control that separate slice of the hardware.
> >
> > What makes this a bit confusing is that for the sake of simplicity (and,
> > I guess, lack of foresight) the original bindings were written in a way
> > to encompass all registers without making that distinction. This worked
> > fine because we've only ever run Linux as host OS where it has access to
> > all those registers.
> >
> > However, when we move to virtualized environments that no longer works.
> >
> > Given the above, we can't read any registers in order to probe whether
> > we run as a guest or not. Trying to access any of the global registers
> > from a VM simply won't work and may crash the system. None of the
> > "channel" registers contain information indicating host vs. guest
> > either.
>
> I don't understand how it differs from what I said - you want to
> indicate that you run in virtualized environment and not all resources
> are accessible.
>
> The device still has the first (global) address, just it is not
> available due to hypervisor.
Yes, and that's a bad thing because there's no way for the device to
know that it can't access the registers. So it will just assume that it
can and try to access them, which would then result in a crash/error.
Thierry
On 30/05/2024 14:48, Thierry Reding wrote:
> On Tue May 28, 2024 at 8:48 AM CEST, Krzysztof Kozlowski wrote:
>> On 24/05/2024 09:36, Thierry Reding wrote:
>>> On Wed May 22, 2024 at 1:29 PM CEST, Krzysztof Kozlowski wrote:
>>>> On 22/05/2024 09:43, Sameer Pujar wrote:
>>>>>
>>>>>
>>>>> On 22-05-2024 12:17, Krzysztof Kozlowski wrote:
>>>>>> On 22/05/2024 07:35, Sameer Pujar wrote:
>>>>>>> On 21-05-2024 17:23, Krzysztof Kozlowski wrote:
>>>>>>>> On 21/05/2024 13:08, Sameer Pujar wrote:
>>>>>>>>> From: Mohan Kumar <[email protected]>
>>>>>>>>>
>>>>>>>>> For Non-Hypervisor mode, Tegra ADMA driver requires the register
>>>>>>>>> resource range to include both global and channel page in the reg
>>>>>>>>> entry. For Hypervisor more, Tegra ADMA driver requires only the
>>>>>>>>> channel page and global page range is not allowed for access.
>>>>>>>>>
>>>>>>>>> Add reg-names DT binding for Hypervisor mode to help driver to
>>>>>>>>> differentiate the config between Hypervisor and Non-Hypervisor
>>>>>>>>> mode of execution.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Mohan Kumar <[email protected]>
>>>>>>>>> Signed-off-by: Sameer Pujar <[email protected]>
>>>>>>>>> ---
>>>>>>>>> .../devicetree/bindings/dma/nvidia,tegra210-adma.yaml | 10 ++++++++++
>>>>>>>>> 1 file changed, 10 insertions(+)
>>>>>>>>>
>>>>>>>>> diff --git a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>>>>>>>>> index 877147e95ecc..ede47f4a3eec 100644
>>>>>>>>> --- a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>>>>>>>>> +++ b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
>>>>>>>>> @@ -29,8 +29,18 @@ properties:
>>>>>>>>> - const: nvidia,tegra186-adma
>>>>>>>>>
>>>>>>>>> reg:
>>>>>>>>> + description: |
>>>>>>>>> + For hypervisor mode, the address range should include a
>>>>>>>>> + ADMA channel page address range, for non-hypervisor mode
>>>>>>>>> + it starts with ADMA base address covering Global and Channel
>>>>>>>>> + page address range.
>>>>>>>>> maxItems: 1
>>>>>>>>>
>>>>>>>>> + reg-names:
>>>>>>>>> + description: only required for Hypervisor mode.
>>>>>>>> This does not work like that. I provide vm entry for non-hypervisor mode
>>>>>>>> and what? You claim it is virtualized?
>>>>>>>>
>>>>>>>> Drop property.
>>>>>>> With 'vm' entry added for hypervisor mode, the 'reg' address range needs
>>>>>>> to be updated to use channel specific region only. This is used to
>>>>>>> inform driver to skip global regions which is taken care by hypervisor.
>>>>>>> This is expected to be used in the scenario where Linux acts as a
>>>>>>> virtual machine (VM). May be the hypervisor mode gives a different
>>>>>>> impression here? Sorry, I did not understand what dropping the property
>>>>>>> exactly means here.
>>>>>> It was imperative. Drop it. Remove it. I provided explanation why.
>>>>>
>>>>> The driver doesn't know if it is operated in a native config or in the
>>>>> hypervisor config based on the 'reg' address range alone. So 'vm' entry
>>>>> with restricted 'reg' range is used to differentiate here for the
>>>>> hypervisor config. Just adding 'vm' entry won't be enough, the 'reg'
>>>>> region must be updated as well to have expected behavior. Not sure how
>>>>> this dependency can be enforced in the schema.
>>>>
>>>> That's not a unusual problem, so please come with a solution for your
>>>> entire subarch. We've been discussing similar topic in terms of SCMI
>>>> controlled resources (see talk on Linaro Connect a week ago:
>>>> https://www.kitefor.events/events/linaro-connect-24/submissions/161 I
>>>> don't know where is recording or slides, see also discussions on mailing
>>>> lists about it), which is not that far away from the problem here. Other
>>>> platforms and maybe nvidia had as well changes in IO space for
>>>> virtualized configuration.
>>>>
>>>> Come with unified approach FOR ALL your devices, not only this one
>>>> (that's kind of basic thing we keep repeating... don't solve only one
>>>> your problem), do not abuse the regular property, because as I said:
>>>> reg-names will be provided as well in non-vm case and then your entire
>>>> logic is wrong. The purpose of reg-names is not to tell whether you have
>>>> or have not virtualized environment.
>>>
>>> This isn't strictly about telling whether this is a virtualized
>>> environment or not. Unfortunately the bindings don't make that very
>>> clear, so let me try to give a bit more background.
>>>
>>> On Tegra devices the register regions associated with a device are
>>> usually split up into 64 KiB chunks.
>>
>> So describing it as one IO region was incorrect from the start and you
>> want to fix it by adding one more incorrect description: making first
>> item meaning two different things. Sorry, that's not a correct way to
>> fix things.
>
> Yes, describing this as one I/O region was incorrect, and in hindsight
> it should have been done differently.
>
> However, I don't think it's correct to describe this as adding one more
> incorrect description. Instead, what this does is add reg-names to
> provide additional context so that the operating system can make the
> necessary decisions as to what is allowed and what isn't.
>
> In the absence of a reg-names property the current definition of the DT
> bindings applies, so it means the region represents the entirety of the
> device's I/O register space. That's one particular use-case for this
> device.
>
> For additional use-cases we can then use reg-names to differentiate
> between what separate regions are and use them accordingly.
>
>> Items are defined, thus first item is always expected to be what the
>> binding already said. Adding reg-names changes nothing, because (as
>> repeated many times) xxx-names is just a helper. Items are already defined.
>
> I don't understand what you're trying to say here. I suppose adding
> reg-names alone indeed doesn't change anything. But the point is that
> once added we can now use these properties, at which point of course
> things change.
>
>>> One of these chunks, usually the first one, is a global region that
>>> contains registers that configure the device as a whole. This is usually
>>> privileged and accessible only to the hypervisor.
>>>
>>> Subsequent regions are meant to be assigned to individual VMs. Often the
>>> regions take the form of "channels", so they are instances of the same
>>> register block and control that separate slice of the hardware.
>>>
>>> What makes this a bit confusing is that for the sake of simplicity (and,
>>> I guess, lack of foresight) the original bindings were written in a way
>>> to encompass all registers without making that distinction. This worked
>>> fine because we've only ever run Linux as host OS where it has access to
>>> all those registers.
>>>
>>> However, when we move to virtualized environments that no longer works.
>>>
>>> Given the above, we can't read any registers in order to probe whether
>>> we run as a guest or not. Trying to access any of the global registers
>>> from a VM simply won't work and may crash the system. None of the
>>> "channel" registers contain information indicating host vs. guest
>>> either.
>>
>> I don't understand how it differs from what I said - you want to
>> indicate that you run in virtualized environment and not all resources
>> are accessible.
>>
>> The device still has the first (global) address, just it is not
>> available due to hypervisor.
>
> Yes, and that's a bad thing because there's no way for the device to
> know that it can't access the registers. So it will just assume that it
> can and try to access them, which would then result in a crash/error.
Different compatible could note that or the global address would be
removed from IO space, although then you need to rely on names and order
is not fixed. I think Rob already proposed different compatible.
This is also the way new Qcom platforms are going (older were using
properties).
However my earlier comment stays on: you will have for sure more cases
like this, so please think upfront and pick unified approach for all
future devices.
Best regards,
Krzysztof
On Fri May 31, 2024 at 9:43 AM CEST, Krzysztof Kozlowski wrote:
> On 30/05/2024 14:48, Thierry Reding wrote:
> > On Tue May 28, 2024 at 8:48 AM CEST, Krzysztof Kozlowski wrote:
> >> On 24/05/2024 09:36, Thierry Reding wrote:
> >>> On Wed May 22, 2024 at 1:29 PM CEST, Krzysztof Kozlowski wrote:
> >>>> On 22/05/2024 09:43, Sameer Pujar wrote:
> >>>>>
> >>>>>
> >>>>> On 22-05-2024 12:17, Krzysztof Kozlowski wrote:
> >>>>>> On 22/05/2024 07:35, Sameer Pujar wrote:
> >>>>>>> On 21-05-2024 17:23, Krzysztof Kozlowski wrote:
> >>>>>>>> On 21/05/2024 13:08, Sameer Pujar wrote:
> >>>>>>>>> From: Mohan Kumar <[email protected]>
> >>>>>>>>>
> >>>>>>>>> For Non-Hypervisor mode, Tegra ADMA driver requires the register
> >>>>>>>>> resource range to include both global and channel page in the reg
> >>>>>>>>> entry. For Hypervisor more, Tegra ADMA driver requires only the
> >>>>>>>>> channel page and global page range is not allowed for access.
> >>>>>>>>>
> >>>>>>>>> Add reg-names DT binding for Hypervisor mode to help driver to
> >>>>>>>>> differentiate the config between Hypervisor and Non-Hypervisor
> >>>>>>>>> mode of execution.
> >>>>>>>>>
> >>>>>>>>> Signed-off-by: Mohan Kumar <[email protected]>
> >>>>>>>>> Signed-off-by: Sameer Pujar <[email protected]>
> >>>>>>>>> ---
> >>>>>>>>> .../devicetree/bindings/dma/nvidia,tegra210-adma.yaml | 10 ++++++++++
> >>>>>>>>> 1 file changed, 10 insertions(+)
> >>>>>>>>>
> >>>>>>>>> diff --git a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> >>>>>>>>> index 877147e95ecc..ede47f4a3eec 100644
> >>>>>>>>> --- a/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> >>>>>>>>> +++ b/Documentation/devicetree/bindings/dma/nvidia,tegra210-adma.yaml
> >>>>>>>>> @@ -29,8 +29,18 @@ properties:
> >>>>>>>>> - const: nvidia,tegra186-adma
> >>>>>>>>>
> >>>>>>>>> reg:
> >>>>>>>>> + description: |
> >>>>>>>>> + For hypervisor mode, the address range should include a
> >>>>>>>>> + ADMA channel page address range, for non-hypervisor mode
> >>>>>>>>> + it starts with ADMA base address covering Global and Channel
> >>>>>>>>> + page address range.
> >>>>>>>>> maxItems: 1
> >>>>>>>>>
> >>>>>>>>> + reg-names:
> >>>>>>>>> + description: only required for Hypervisor mode.
> >>>>>>>> This does not work like that. I provide vm entry for non-hypervisor mode
> >>>>>>>> and what? You claim it is virtualized?
> >>>>>>>>
> >>>>>>>> Drop property.
> >>>>>>> With 'vm' entry added for hypervisor mode, the 'reg' address range needs
> >>>>>>> to be updated to use channel specific region only. This is used to
> >>>>>>> inform driver to skip global regions which is taken care by hypervisor.
> >>>>>>> This is expected to be used in the scenario where Linux acts as a
> >>>>>>> virtual machine (VM). May be the hypervisor mode gives a different
> >>>>>>> impression here? Sorry, I did not understand what dropping the property
> >>>>>>> exactly means here.
> >>>>>> It was imperative. Drop it. Remove it. I provided explanation why.
> >>>>>
> >>>>> The driver doesn't know if it is operated in a native config or in the
> >>>>> hypervisor config based on the 'reg' address range alone. So 'vm' entry
> >>>>> with restricted 'reg' range is used to differentiate here for the
> >>>>> hypervisor config. Just adding 'vm' entry won't be enough, the 'reg'
> >>>>> region must be updated as well to have expected behavior. Not sure how
> >>>>> this dependency can be enforced in the schema.
> >>>>
> >>>> That's not a unusual problem, so please come with a solution for your
> >>>> entire subarch. We've been discussing similar topic in terms of SCMI
> >>>> controlled resources (see talk on Linaro Connect a week ago:
> >>>> https://www.kitefor.events/events/linaro-connect-24/submissions/161 I
> >>>> don't know where is recording or slides, see also discussions on mailing
> >>>> lists about it), which is not that far away from the problem here. Other
> >>>> platforms and maybe nvidia had as well changes in IO space for
> >>>> virtualized configuration.
> >>>>
> >>>> Come with unified approach FOR ALL your devices, not only this one
> >>>> (that's kind of basic thing we keep repeating... don't solve only one
> >>>> your problem), do not abuse the regular property, because as I said:
> >>>> reg-names will be provided as well in non-vm case and then your entire
> >>>> logic is wrong. The purpose of reg-names is not to tell whether you have
> >>>> or have not virtualized environment.
> >>>
> >>> This isn't strictly about telling whether this is a virtualized
> >>> environment or not. Unfortunately the bindings don't make that very
> >>> clear, so let me try to give a bit more background.
> >>>
> >>> On Tegra devices the register regions associated with a device are
> >>> usually split up into 64 KiB chunks.
> >>
> >> So describing it as one IO region was incorrect from the start and you
> >> want to fix it by adding one more incorrect description: making first
> >> item meaning two different things. Sorry, that's not a correct way to
> >> fix things.
> >
> > Yes, describing this as one I/O region was incorrect, and in hindsight
> > it should have been done differently.
> >
> > However, I don't think it's correct to describe this as adding one more
> > incorrect description. Instead, what this does is add reg-names to
> > provide additional context so that the operating system can make the
> > necessary decisions as to what is allowed and what isn't.
> >
> > In the absence of a reg-names property the current definition of the DT
> > bindings applies, so it means the region represents the entirety of the
> > device's I/O register space. That's one particular use-case for this
> > device.
> >
> > For additional use-cases we can then use reg-names to differentiate
> > between what separate regions are and use them accordingly.
> >
> >> Items are defined, thus first item is always expected to be what the
> >> binding already said. Adding reg-names changes nothing, because (as
> >> repeated many times) xxx-names is just a helper. Items are already defined.
> >
> > I don't understand what you're trying to say here. I suppose adding
> > reg-names alone indeed doesn't change anything. But the point is that
> > once added we can now use these properties, at which point of course
> > things change.
> >
> >>> One of these chunks, usually the first one, is a global region that
> >>> contains registers that configure the device as a whole. This is usually
> >>> privileged and accessible only to the hypervisor.
> >>>
> >>> Subsequent regions are meant to be assigned to individual VMs. Often the
> >>> regions take the form of "channels", so they are instances of the same
> >>> register block and control that separate slice of the hardware.
> >>>
> >>> What makes this a bit confusing is that for the sake of simplicity (and,
> >>> I guess, lack of foresight) the original bindings were written in a way
> >>> to encompass all registers without making that distinction. This worked
> >>> fine because we've only ever run Linux as host OS where it has access to
> >>> all those registers.
> >>>
> >>> However, when we move to virtualized environments that no longer works.
> >>>
> >>> Given the above, we can't read any registers in order to probe whether
> >>> we run as a guest or not. Trying to access any of the global registers
> >>> from a VM simply won't work and may crash the system. None of the
> >>> "channel" registers contain information indicating host vs. guest
> >>> either.
> >>
> >> I don't understand how it differs from what I said - you want to
> >> indicate that you run in virtualized environment and not all resources
> >> are accessible.
> >>
> >> The device still has the first (global) address, just it is not
> >> available due to hypervisor.
> >
> > Yes, and that's a bad thing because there's no way for the device to
> > know that it can't access the registers. So it will just assume that it
> > can and try to access them, which would then result in a crash/error.
>
> Different compatible could note that or the global address would be
> removed from IO space, although then you need to rely on names and order
> is not fixed. I think Rob already proposed different compatible.
>
> This is also the way new Qcom platforms are going (older were using
> properties).
>
> However my earlier comment stays on: you will have for sure more cases
> like this, so please think upfront and pick unified approach for all
> future devices.
We already have. In fact we already have a few devices (host1x[0] and
MGBE[1]) where a similar path was chosen. Unification with those is why
we're proposing this.
This also applies to the memory controller SID bindings update that we
proposed a little while ago.
Thierry
[0]:
Documentation/devicetree/bindings/display/tegra/nvidia,tegra20-host1x.yaml
[1]: Documentation/devicetree/bindings/net/nvidia,tegra234-mgbe.yaml