2021-10-11 16:59:51

by Hector Martin

[permalink] [raw]
Subject: [RFC PATCH 0/9] Apple SoC CPU P-state switching

Hi folks, here's a first attempt at cpufreq support for the Apple M1.
I'd appreciate any feedback on this approach.

The hardware in these SoCs is very simple: you just poke a single
register to change the performance state of a CPU cluster. There's
some init required on older firmware versions, but we're going to
say that's the bootloader's responsibility. This keeps the driver
nice and simple and generic and likely to work on future SoCs too.

However, there's a catch: the memory controller config should also be
modified when higher clock states are engaged on the P-cores, to
get optimal performance.

This series implements this using two drivers, on top of the existing
cpufreq-dt infrastructure. The cpu clock driver represents the p-state
switching as if it were a standard clock driver, so it can work with
cpufreq-dt. To do this, it also needs access to the OPP table, so it can
map the incoming clock frequences back to P-State index numbers, so that
is present in the binding. This might be a bit strange, since the same
OPP table is referenced by the CPUs themselves, and by the clocks driver
that provides the actual switching for them...

The memory controller stuff is implemented as a genpd provider that
exposes two performance states that the CPU OPP tables can depend on.
Unfortunately, the cpufreq path doesn't work properly for this, since
the CPUs aren't typical devices participating in runtime-pm. So instead
I opted to put that logic in the clock driver, which means it gets a
power-domains reference to the memory controller. This required a hack
to the OPP core so that it wouldn't complain about the missing parent
domain when evaluating the OPPs in the context of the CPUs...

The actual memory controller config is two magic numbers per performance
state. I'd love to find out what they do, but this seems unlikely
without docs or a deep memory performance analysis expedition... so I
think we're going to have to settle for this way, at least for now. If
things become better understood in the future, we can modify the binding
and keep the driver backwards-compatible with old DTs at least.

I did benchmark the CPU p-state switching, so the latency numbers there
have been experimentally measured. The CPU capacity numbers are also
based on real benchmarks (oddly enough, Dhrystone was a big outlier
here that was not representative of everything else, so we didn't use
it).

Patches:
#1: MAINTAINERS updates, split out so this can go through the SoC
tree so we can spare all the subsystem maintainers the merge
conflicts, since we have a bunch of other changes here going on
in parallel.
#2-3: DT bindings
#4: The aforementioned hack for the OPP core
#5: Add of_genpd_add_provider_simple_noclk()
#6: Memory controller driver
#7: CPU p-state clock driver
#8: Add some deps for ARCH_APPLE
#9: DT updates (for asahi-soc tree)

Hector Martin (9):
MAINTAINERS: apple: Add apple-mcc and clk-apple-cluster paths
dt-bindings: memory-controller: Add apple,mcc binding
dt-bindings: clock: Add apple,cluster-clk binding
opp: core: Don't warn if required OPP device does not exist
PM: domains: Add of_genpd_add_provider_simple_noclk()
memory: apple: Add apple-mcc driver to manage MCC perf in Apple SoCs
clk: apple: Add clk-apple-cluster driver to manage CPU p-states
arm64: apple: Select MEMORY and APPLE_MCC
arm64: apple: Add CPU frequency scaling support for t8103

.../bindings/clock/apple,cluster-clk.yaml | 115 ++++++++
.../memory-controllers/apple,mcc.yaml | 80 ++++++
.../opp/apple,mcc-operating-points.yaml | 62 +++++
MAINTAINERS | 5 +
arch/arm64/Kconfig.platforms | 2 +
arch/arm64/boot/dts/apple/t8103.dtsi | 255 +++++++++++++++++-
drivers/base/power/domain.c | 39 ++-
drivers/clk/Kconfig | 9 +
drivers/clk/Makefile | 1 +
drivers/clk/clk-apple-cluster.c | 184 +++++++++++++
drivers/memory/Kconfig | 9 +
drivers/memory/Makefile | 1 +
drivers/memory/apple-mcc.c | 130 +++++++++
drivers/opp/core.c | 5 +-
include/linux/pm_domain.h | 8 +
15 files changed, 887 insertions(+), 18 deletions(-)
create mode 100644 Documentation/devicetree/bindings/clock/apple,cluster-clk.yaml
create mode 100644 Documentation/devicetree/bindings/memory-controllers/apple,mcc.yaml
create mode 100644 Documentation/devicetree/bindings/opp/apple,mcc-operating-points.yaml
create mode 100644 drivers/clk/clk-apple-cluster.c
create mode 100644 drivers/memory/apple-mcc.c

--
2.33.0


2021-10-11 17:00:09

by Hector Martin

[permalink] [raw]
Subject: [RFC PATCH 2/9] dt-bindings: memory-controller: Add apple,mcc binding

This device represents the memory controller in Apple SoCs, and is
chiefly in charge of adjusting performance characteristics according to
system demand.

Signed-off-by: Hector Martin <[email protected]>
---
.../memory-controllers/apple,mcc.yaml | 80 +++++++++++++++++++
.../opp/apple,mcc-operating-points.yaml | 62 ++++++++++++++
2 files changed, 142 insertions(+)
create mode 100644 Documentation/devicetree/bindings/memory-controllers/apple,mcc.yaml
create mode 100644 Documentation/devicetree/bindings/opp/apple,mcc-operating-points.yaml

diff --git a/Documentation/devicetree/bindings/memory-controllers/apple,mcc.yaml b/Documentation/devicetree/bindings/memory-controllers/apple,mcc.yaml
new file mode 100644
index 000000000000..0774f10e65ed
--- /dev/null
+++ b/Documentation/devicetree/bindings/memory-controllers/apple,mcc.yaml
@@ -0,0 +1,80 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/memory-controllers/apple,mcc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Apple SoC MCC memory controller performance controls
+
+maintainers:
+ - Hector Martin <[email protected]>
+
+description: |
+ Apple SoCs contain a multichannel memory controller that can have its
+ configuration changed to adjust to changing performance requirements from
+ the rest of the SoC. This node represents the controller and provides a
+ power domain provider that downstream devices can use to adjust the memory
+ controller performance level.
+
+properties:
+ compatible:
+ items:
+ - enum:
+ - apple,t8103-mcc
+ - const: apple,mcc
+
+ reg:
+ maxItems: 1
+
+ "#power-domain-cells":
+ const: 0
+
+ operating-points-v2:
+ $ref: /schemas/types.yaml#/definitions/phandle-array
+ description:
+ A reference to the OPP table describing the memory controller performance
+ levels. Each OPP node should contain an `apple,memory-perf-config`
+ property that contains the configuration values for that performance
+ level.
+
+ apple,num-channels:
+ $ref: /schemas/types.yaml#/definitions/uint32
+ description:
+ The number of memory channels in use.
+
+required:
+ - compatible
+ - reg
+ - "#power-domain-cells"
+ - operating-points-v2
+ - apple,num-channels
+
+additionalProperties: false
+
+examples:
+ # See clock/apple,cluster-clock.yaml for an example of downstream usage.
+ - |
+ mcc_opp: opp-table-2 {
+ compatible = "operating-points-v2";
+
+ mcc_lowperf: opp0 {
+ opp-level = <0>;
+ apple,memory-perf-config = <0x813057f 0x1800180>;
+ };
+ mcc_highperf: opp1 {
+ opp-level = <1>;
+ apple,memory-perf-config = <0x133 0x55555340>;
+ };
+ };
+ soc {
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ mcc: memory-controller@200200000 {
+ compatible = "apple,t8103-mcc", "apple,mcc";
+ #power-domain-cells = <0>;
+ reg = <0x2 0x200000 0x0 0x200000>;
+ operating-points-v2 = <&mcc_opp>;
+ apple,num-channels = <8>;
+ };
+ };
diff --git a/Documentation/devicetree/bindings/opp/apple,mcc-operating-points.yaml b/Documentation/devicetree/bindings/opp/apple,mcc-operating-points.yaml
new file mode 100644
index 000000000000..babf27841bb7
--- /dev/null
+++ b/Documentation/devicetree/bindings/opp/apple,mcc-operating-points.yaml
@@ -0,0 +1,62 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/opp/apple,mcc-operating-points.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Apple SoC memory controller OPP bindings
+
+maintainers:
+ - Hector Martin <[email protected]>
+
+description: |
+ Apple SoCs can have their memory controller performance adjusted depending on
+ system requirements. These performance states are represented by specific
+ memory controller register values. The apple-mcc driver uses these values
+ to change the MCC performance.
+
+allOf:
+ - $ref: opp-v2-base.yaml#
+
+properties:
+ compatible:
+ const: apple,mcc-operating-points
+
+required:
+ - compatible
+
+patternProperties:
+ "opp[0-9]+":
+ type: object
+
+ properties:
+ opp-level: true
+ apple,memory-perf-config:
+ $ref: /schemas/types.yaml#/definitions/uint32-array
+ description: |
+ A pair of register values used to configure this performance state.
+ minItems: 2
+ maxItems: 2
+
+ required:
+ - opp-level
+ - apple,memory-perf-config
+
+ unevaluatedProperties: false
+
+additionalProperties: false
+
+examples:
+ - |
+ mcc_opp: opp-table-2 {
+ compatible = "operating-points-v2";
+
+ mcc_lowperf: opp0 {
+ opp-level = <0>;
+ apple,memory-perf-config = <0x813057f 0x1800180>;
+ };
+ mcc_highperf: opp1 {
+ opp-level = <1>;
+ apple,memory-perf-config = <0x133 0x55555340>;
+ };
+ };
--
2.33.0

2021-10-11 17:01:01

by Hector Martin

[permalink] [raw]
Subject: [RFC PATCH 3/9] dt-bindings: clock: Add apple,cluster-clk binding

This device represents the CPU performance state switching mechanism as
a clock controller, to be used with the standard cpufreq-dt
infrastructure.

Signed-off-by: Hector Martin <[email protected]>
---
.../bindings/clock/apple,cluster-clk.yaml | 115 ++++++++++++++++++
1 file changed, 115 insertions(+)
create mode 100644 Documentation/devicetree/bindings/clock/apple,cluster-clk.yaml

diff --git a/Documentation/devicetree/bindings/clock/apple,cluster-clk.yaml b/Documentation/devicetree/bindings/clock/apple,cluster-clk.yaml
new file mode 100644
index 000000000000..9a8b863dadc0
--- /dev/null
+++ b/Documentation/devicetree/bindings/clock/apple,cluster-clk.yaml
@@ -0,0 +1,115 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/clock/apple,cluster-clk.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: CPU cluster frequency scaling for Apple SoCs
+
+maintainers:
+ - Hector Martin <[email protected]>
+
+description: |
+ Apple SoCs control CPU cluster frequencies by using a performance state
+ index. This node represents the feature as a clock controller, and uses
+ a reference to the CPU OPP table to translate clock frequencies into
+ performance states. This allows the CPUs to use the standard cpufreq-dt
+ mechanism for frequency scaling.
+
+properties:
+ compatible:
+ items:
+ - enum:
+ - apple,t8103-cluster-clk
+ - const: apple,cluster-clk
+
+ reg:
+ maxItems: 1
+
+ '#clock-cells':
+ const: 0
+
+ operating-points-v2:
+ $ref: /schemas/types.yaml#/definitions/phandle-array
+ description:
+ A reference to the OPP table used for the CPU cluster controlled by this
+ device instance. The table should contain an `opp-level` property for
+ every OPP, which represents the p-state index used by the hardware to
+ represent this performance level.
+
+ OPPs may also have a `required-opps` property (see power-domains).
+
+ power-domains:
+ maxItems: 1
+ description:
+ An optional reference to a power domain provider that links its
+ performance state to the CPU cluster performance state. This is typically
+ a memory controller. If set, the `required-opps` property in the CPU
+ frequency OPP nodes will be used to change the performance state of this
+ provider state in tandem with CPU frequency changes.
+
+required:
+ - compatible
+ - reg
+ - '#clock-cells'
+ - operating-points-v2
+
+additionalProperties: false
+
+
+examples:
+ - |
+ pcluster_opp: opp-table-1 {
+ compatible = "operating-points-v2";
+ opp-shared;
+
+ opp01 {
+ opp-hz = /bits/ 64 <600000000>;
+ opp-microvolt = <781000>;
+ opp-level = <1>;
+ clock-latency-ns = <8000>;
+ required-opps = <&mcc_lowperf>;
+ };
+ /* intermediate p-states omitted */
+ opp15 {
+ opp-hz = /bits/ 64 <3204000000>;
+ opp-microvolt = <1081000>;
+ opp-level = <15>;
+ clock-latency-ns = <56000>;
+ required-opps = <&mcc_highperf>;
+ };
+ };
+
+ mcc_opp: opp-table-2 {
+ compatible = "operating-points-v2";
+
+ mcc_lowperf: opp0 {
+ opp-level = <0>;
+ apple,memory-perf-config = <0x813057f 0x1800180>;
+ };
+ mcc_highperf: opp1 {
+ opp-level = <1>;
+ apple,memory-perf-config = <0x133 0x55555340>;
+ };
+ };
+
+ soc {
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ mcc: memory-controller@200200000 {
+ compatible = "apple,t8103-mcc", "apple,mcc";
+ #power-domain-cells = <0>;
+ reg = <0x2 0x200000 0x0 0x200000>;
+ operating-points-v2 = <&mcc_opp>;
+ apple,num-channels = <8>;
+ };
+
+ clk_pcluster: clock-controller@211e20000 {
+ compatible = "apple,t8103-cluster-clk", "apple,cluster-clk";
+ #clock-cells = <0>;
+ reg = <0x2 0x11e20000 0x0 0x4000>;
+ operating-points-v2 = <&pcluster_opp>;
+ power-domains = <&mcc>;
+ };
+ };
--
2.33.0

2021-10-11 17:01:05

by Hector Martin

[permalink] [raw]
Subject: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

When required-opps is used in CPU OPP tables, there is no parent power
domain to drive it. Squelch this error, to allow a clock driver to
handle this directly instead.

Signed-off-by: Hector Martin <[email protected]>
---
drivers/opp/core.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/opp/core.c b/drivers/opp/core.c
index 04b4691a8aac..89e616721f70 100644
--- a/drivers/opp/core.c
+++ b/drivers/opp/core.c
@@ -873,12 +873,13 @@ static int _set_required_opp(struct device *dev, struct device *pd_dev,
return 0;

ret = dev_pm_genpd_set_performance_state(pd_dev, pstate);
- if (ret) {
+ if (ret && ret != -ENODEV) {
dev_err(dev, "Failed to set performance rate of %s: %d (%d)\n",
dev_name(pd_dev), pstate, ret);
+ return ret;
}

- return ret;
+ return 0;
}

/* This is only called for PM domain for now */
--
2.33.0

2021-10-12 03:24:19

by Viresh Kumar

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On 12-10-21, 01:57, Hector Martin wrote:
> When required-opps is used in CPU OPP tables, there is no parent power
> domain to drive it. Squelch this error, to allow a clock driver to
> handle this directly instead.
>
> Signed-off-by: Hector Martin <[email protected]>
> ---
> drivers/opp/core.c | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/opp/core.c b/drivers/opp/core.c
> index 04b4691a8aac..89e616721f70 100644
> --- a/drivers/opp/core.c
> +++ b/drivers/opp/core.c
> @@ -873,12 +873,13 @@ static int _set_required_opp(struct device *dev, struct device *pd_dev,
> return 0;
>
> ret = dev_pm_genpd_set_performance_state(pd_dev, pstate);
> - if (ret) {
> + if (ret && ret != -ENODEV) {
> dev_err(dev, "Failed to set performance rate of %s: %d (%d)\n",
> dev_name(pd_dev), pstate, ret);
> + return ret;
> }
>
> - return ret;
> + return 0;
> }
>
> /* This is only called for PM domain for now */

I am not sure why you need this, since _set_required_opps() has this check:

if (unlikely(!required_opp_tables[0]->is_genpd)) {
dev_err(dev, "required-opps don't belong to a genpd\n");
return -ENOENT;
}

--
viresh

2021-10-12 05:36:51

by Hector Martin

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On 12/10/2021 12.21, Viresh Kumar wrote:
> I am not sure why you need this, since _set_required_opps() has this check:
>
> if (unlikely(!required_opp_tables[0]->is_genpd)) {
> dev_err(dev, "required-opps don't belong to a genpd\n");
> return -ENOENT;
> }
>

The table *is* assigned to a genpd (the memory controller), it's just
that that genpd isn't actually a parent of the CPU device. Without the
patch you end up with:

[ 3.040060] cpu cpu4: Failed to set performance rate of cpu4: 0 (-19)
[ 3.042881] cpu cpu4: Failed to set required opps: -19
[ 3.045508] cpufreq: __target_index: Failed to change cpu frequency: -19


--
Hector Martin ([email protected])
Public Key: https://mrcn.st/pub

2021-10-12 05:53:33

by Viresh Kumar

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On 12-10-21, 14:34, Hector Martin wrote:
> The table *is* assigned to a genpd (the memory controller), it's just that
> that genpd isn't actually a parent of the CPU device. Without the patch you
> end up with:
>
> [ 3.040060] cpu cpu4: Failed to set performance rate of cpu4: 0 (-19)
> [ 3.042881] cpu cpu4: Failed to set required opps: -19
> [ 3.045508] cpufreq: __target_index: Failed to change cpu frequency: -19

Hmm, Saravana and Sibi were working on a similar problem earlier and decided to
solve this using devfreq instead. Don't remember the exact series which got
merged for this, Sibi ?

If this part fails, how do you actually set the performance state of the memory
controller's genpd ?

--
viresh

2021-10-12 06:00:52

by Hector Martin

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On 12/10/2021 14.51, Viresh Kumar wrote:
> On 12-10-21, 14:34, Hector Martin wrote:
>> The table *is* assigned to a genpd (the memory controller), it's just that
>> that genpd isn't actually a parent of the CPU device. Without the patch you
>> end up with:
>>
>> [ 3.040060] cpu cpu4: Failed to set performance rate of cpu4: 0 (-19)
>> [ 3.042881] cpu cpu4: Failed to set required opps: -19
>> [ 3.045508] cpufreq: __target_index: Failed to change cpu frequency: -19
>
> Hmm, Saravana and Sibi were working on a similar problem earlier and decided to
> solve this using devfreq instead. Don't remember the exact series which got
> merged for this, Sibi ?
>
> If this part fails, how do you actually set the performance state of the memory
> controller's genpd ?

The clock controller has the genpd as an actual power-domain parent, so
it does it instead. From patch #7:

> + if (cluster->has_pd)
> + dev_pm_genpd_set_performance_state(cluster->dev,
> + dev_pm_opp_get_required_pstate(opp, 0));
> +

This is arguably not entirely representative of how the hardware works,
since technically the cluster switching couldn't care less what the
memory controller is doing; it's a soft dependency, states that should
be switched together but are not interdependent (in fact, the clock code
does this unconditionally after the CPU p-state change, regardless of
whether we're shifting up or down; this is, FWIW, the same order macOS
uses, and it clearly doesn't matter which way you do it).

--
Hector Martin ([email protected])
Public Key: https://mrcn.st/pub

2021-10-12 08:50:09

by Krzysztof Kozlowski

[permalink] [raw]
Subject: Re: [RFC PATCH 2/9] dt-bindings: memory-controller: Add apple,mcc binding

On 11/10/2021 18:57, Hector Martin wrote:
> This device represents the memory controller in Apple SoCs, and is
> chiefly in charge of adjusting performance characteristics according to
> system demand.
>
> Signed-off-by: Hector Martin <[email protected]>
> ---
> .../memory-controllers/apple,mcc.yaml | 80 +++++++++++++++++++
> .../opp/apple,mcc-operating-points.yaml | 62 ++++++++++++++
> 2 files changed, 142 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/memory-controllers/apple,mcc.yaml
> create mode 100644 Documentation/devicetree/bindings/opp/apple,mcc-operating-points.yaml
>
> diff --git a/Documentation/devicetree/bindings/memory-controllers/apple,mcc.yaml b/Documentation/devicetree/bindings/memory-controllers/apple,mcc.yaml
> new file mode 100644
> index 000000000000..0774f10e65ed
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/memory-controllers/apple,mcc.yaml
> @@ -0,0 +1,80 @@
> +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +%YAML 1.2
> +---
> +$id: http://devicetree.org/schemas/memory-controllers/apple,mcc.yaml#
> +$schema: http://devicetree.org/meta-schemas/core.yaml#
> +
> +title: Apple SoC MCC memory controller performance controls
> +
> +maintainers:
> + - Hector Martin <[email protected]>
> +
> +description: |
> + Apple SoCs contain a multichannel memory controller that can have its
> + configuration changed to adjust to changing performance requirements from
> + the rest of the SoC. This node represents the controller and provides a
> + power domain provider that downstream devices can use to adjust the memory
> + controller performance level.
> +
> +properties:
> + compatible:
> + items:
> + - enum:
> + - apple,t8103-mcc
> + - const: apple,mcc
> +
> + reg:
> + maxItems: 1
> +
> + "#power-domain-cells":
> + const: 0
> +
> + operating-points-v2:
> + $ref: /schemas/types.yaml#/definitions/phandle-array
> + description:
> + A reference to the OPP table describing the memory controller performance
> + levels. Each OPP node should contain an `apple,memory-perf-config`
> + property that contains the configuration values for that performance
> + level.
> +
> + apple,num-channels:
> + $ref: /schemas/types.yaml#/definitions/uint32
> + description:
> + The number of memory channels in use.
> +
> +required:
> + - compatible
> + - reg
> + - "#power-domain-cells"
> + - operating-points-v2
> + - apple,num-channels
> +
> +additionalProperties: false
> +
> +examples:
> + # See clock/apple,cluster-clock.yaml for an example of downstream usage.
> + - |
> + mcc_opp: opp-table-2 {
> + compatible = "operating-points-v2";

apple,mcc-operating-points?

> +
> + mcc_lowperf: opp0 {
> + opp-level = <0>;
> + apple,memory-perf-config = <0x813057f 0x1800180>;
> + };
> + mcc_highperf: opp1 {
> + opp-level = <1>;
> + apple,memory-perf-config = <0x133 0x55555340>;
> + };
> + };
> + soc {
> + #address-cells = <2>;
> + #size-cells = <2>;
> +
> + mcc: memory-controller@200200000 {
> + compatible = "apple,t8103-mcc", "apple,mcc";
> + #power-domain-cells = <0>;
> + reg = <0x2 0x200000 0x0 0x200000>;
> + operating-points-v2 = <&mcc_opp>;
> + apple,num-channels = <8>;
> + };
> + };
> diff --git a/Documentation/devicetree/bindings/opp/apple,mcc-operating-points.yaml b/Documentation/devicetree/bindings/opp/apple,mcc-operating-points.yaml
> new file mode 100644
> index 000000000000..babf27841bb7
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/opp/apple,mcc-operating-points.yaml
> @@ -0,0 +1,62 @@
> +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +%YAML 1.2
> +---
> +$id: http://devicetree.org/schemas/opp/apple,mcc-operating-points.yaml#
> +$schema: http://devicetree.org/meta-schemas/core.yaml#
> +
> +title: Apple SoC memory controller OPP bindings
> +
> +maintainers:
> + - Hector Martin <[email protected]>
> +
> +description: |
> + Apple SoCs can have their memory controller performance adjusted depending on
> + system requirements. These performance states are represented by specific
> + memory controller register values. The apple-mcc driver uses these values
> + to change the MCC performance.
> +
> +allOf:
> + - $ref: opp-v2-base.yaml#
> +
> +properties:
> + compatible:
> + const: apple,mcc-operating-points
> +
> +required:
> + - compatible
> +
> +patternProperties:
> + "opp[0-9]+":
> + type: object
> +
> + properties:
> + opp-level: true

You don't need to mention it.

> + apple,memory-perf-config:
> + $ref: /schemas/types.yaml#/definitions/uint32-array
> + description: |
> + A pair of register values used to configure this performance state.
> + minItems: 2
> + maxItems: 2
> +
> + required:
> + - opp-level
> + - apple,memory-perf-config
> +
> + unevaluatedProperties: false
> +
> +additionalProperties: false
> +
> +examples:
> + - |
> + mcc_opp: opp-table-2 {
> + compatible = "operating-points-v2";

Different compatible.

> +
> + mcc_lowperf: opp0 {
> + opp-level = <0>;
> + apple,memory-perf-config = <0x813057f 0x1800180>;
> + };
> + mcc_highperf: opp1 {
> + opp-level = <1>;
> + apple,memory-perf-config = <0x133 0x55555340>;
> + };
> + };
>


Best regards,
Krzysztof

2021-10-12 08:55:34

by Krzysztof Kozlowski

[permalink] [raw]
Subject: Re: [RFC PATCH 3/9] dt-bindings: clock: Add apple,cluster-clk binding

On 11/10/2021 18:57, Hector Martin wrote:
> This device represents the CPU performance state switching mechanism as
> a clock controller, to be used with the standard cpufreq-dt
> infrastructure.
>
> Signed-off-by: Hector Martin <[email protected]>
> ---
> .../bindings/clock/apple,cluster-clk.yaml | 115 ++++++++++++++++++
> 1 file changed, 115 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/clock/apple,cluster-clk.yaml
>
> diff --git a/Documentation/devicetree/bindings/clock/apple,cluster-clk.yaml b/Documentation/devicetree/bindings/clock/apple,cluster-clk.yaml
> new file mode 100644
> index 000000000000..9a8b863dadc0
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/clock/apple,cluster-clk.yaml
> @@ -0,0 +1,115 @@
> +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +%YAML 1.2
> +---
> +$id: http://devicetree.org/schemas/clock/apple,cluster-clk.yaml#
> +$schema: http://devicetree.org/meta-schemas/core.yaml#
> +
> +title: CPU cluster frequency scaling for Apple SoCs
> +
> +maintainers:
> + - Hector Martin <[email protected]>
> +
> +description: |
> + Apple SoCs control CPU cluster frequencies by using a performance state
> + index. This node represents the feature as a clock controller, and uses
> + a reference to the CPU OPP table to translate clock frequencies into
> + performance states. This allows the CPUs to use the standard cpufreq-dt
> + mechanism for frequency scaling.
> +
> +properties:
> + compatible:
> + items:
> + - enum:
> + - apple,t8103-cluster-clk
> + - const: apple,cluster-clk
> +
> + reg:
> + maxItems: 1
> +
> + '#clock-cells':
> + const: 0
> +
> + operating-points-v2:
> + $ref: /schemas/types.yaml#/definitions/phandle-array
> + description:
> + A reference to the OPP table used for the CPU cluster controlled by this
> + device instance. The table should contain an `opp-level` property for
> + every OPP, which represents the p-state index used by the hardware to
> + represent this performance level.
> +
> + OPPs may also have a `required-opps` property (see power-domains).
> +
> + power-domains:
> + maxItems: 1
> + description:
> + An optional reference to a power domain provider that links its
> + performance state to the CPU cluster performance state. This is typically
> + a memory controller. If set, the `required-opps` property in the CPU
> + frequency OPP nodes will be used to change the performance state of this
> + provider state in tandem with CPU frequency changes.
> +
> +required:
> + - compatible
> + - reg
> + - '#clock-cells'
> + - operating-points-v2
> +
> +additionalProperties: false
> +
> +

One line break.

> +examples:
> + - |
> + pcluster_opp: opp-table-1 {
> + compatible = "operating-points-v2";
> + opp-shared;
> +
> + opp01 {
> + opp-hz = /bits/ 64 <600000000>;
> + opp-microvolt = <781000>;
> + opp-level = <1>;
> + clock-latency-ns = <8000>;
> + required-opps = <&mcc_lowperf>;
> + };
> + /* intermediate p-states omitted */
> + opp15 {
> + opp-hz = /bits/ 64 <3204000000>;
> + opp-microvolt = <1081000>;
> + opp-level = <15>;
> + clock-latency-ns = <56000>;
> + required-opps = <&mcc_highperf>;
> + };
> + };
> +
> + mcc_opp: opp-table-2 {
> + compatible = "operating-points-v2";

Wrong compatible.

> +
> + mcc_lowperf: opp0 {
> + opp-level = <0>;
> + apple,memory-perf-config = <0x813057f 0x1800180>;
> + };
> + mcc_highperf: opp1 {
> + opp-level = <1>;
> + apple,memory-perf-config = <0x133 0x55555340>;
> + };
> + };
> +
> + soc {
> + #address-cells = <2>;
> + #size-cells = <2>;
> +
> + mcc: memory-controller@200200000 {
> + compatible = "apple,t8103-mcc", "apple,mcc";
> + #power-domain-cells = <0>;
> + reg = <0x2 0x200000 0x0 0x200000>;
> + operating-points-v2 = <&mcc_opp>;
> + apple,num-channels = <8>;
> + };
> +
> + clk_pcluster: clock-controller@211e20000 {
> + compatible = "apple,t8103-cluster-clk", "apple,cluster-clk";
> + #clock-cells = <0>;
> + reg = <0x2 0x11e20000 0x0 0x4000>;
> + operating-points-v2 = <&pcluster_opp>;
> + power-domains = <&mcc>;
> + };
> + };
>


Best regards,
Krzysztof

2021-10-12 09:27:17

by Viresh Kumar

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On 12-10-21, 14:57, Hector Martin wrote:
> On 12/10/2021 14.51, Viresh Kumar wrote:
> > On 12-10-21, 14:34, Hector Martin wrote:
> > > The table *is* assigned to a genpd (the memory controller), it's just that
> > > that genpd isn't actually a parent of the CPU device. Without the patch you
> > > end up with:
> > >
> > > [ 3.040060] cpu cpu4: Failed to set performance rate of cpu4: 0 (-19)
> > > [ 3.042881] cpu cpu4: Failed to set required opps: -19
> > > [ 3.045508] cpufreq: __target_index: Failed to change cpu frequency: -19
> >
> > Hmm, Saravana and Sibi were working on a similar problem earlier and decided to
> > solve this using devfreq instead. Don't remember the exact series which got
> > merged for this, Sibi ?
> >
> > If this part fails, how do you actually set the performance state of the memory
> > controller's genpd ?
>
> The clock controller has the genpd as an actual power-domain parent, so it
> does it instead. From patch #7:
>
> > + if (cluster->has_pd)
> > + dev_pm_genpd_set_performance_state(cluster->dev,
> > + dev_pm_opp_get_required_pstate(opp, 0));
> > +
>
> This is arguably not entirely representative of how the hardware works,
> since technically the cluster switching couldn't care less what the memory
> controller is doing; it's a soft dependency, states that should be switched
> together but are not interdependent (in fact, the clock code does this
> unconditionally after the CPU p-state change, regardless of whether we're
> shifting up or down; this is, FWIW, the same order macOS uses, and it
> clearly doesn't matter which way you do it).

Yeah, I understand what you are doing. But the current patch is
incorrect in the sense that it can cause a bug on other platforms. To
make this work, you should rather set this genpd as parent of CPU
devices (which are doing anyway since you are updating them with CPU's
DVFS). With that the clk driver won't be required to do the magic
behind the scene.

--
viresh

2021-10-12 09:34:13

by Viresh Kumar

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On 12-10-21, 18:31, Hector Martin "marcan" wrote:
> That doesn't work, though, because the CPUs aren't normal devices
> with runtime-pm. That was the first thing I tried :).

What's the exact problem with runtime PM here ?

> If you think this *should* be made to work instead then I can try that.

--
viresh

2021-10-12 09:35:48

by Hector Martin

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist



On 2021年10月12日 18:26:03 JST, Viresh Kumar <[email protected]> wrote:
>On 12-10-21, 14:57, Hector Martin wrote:
>>
>> This is arguably not entirely representative of how the hardware works,
>> since technically the cluster switching couldn't care less what the memory
>> controller is doing; it's a soft dependency, states that should be switched
>> together but are not interdependent (in fact, the clock code does this
>> unconditionally after the CPU p-state change, regardless of whether we're
>> shifting up or down; this is, FWIW, the same order macOS uses, and it
>> clearly doesn't matter which way you do it).
>
>Yeah, I understand what you are doing. But the current patch is
>incorrect in the sense that it can cause a bug on other platforms. To
>make this work, you should rather set this genpd as parent of CPU
>devices (which are doing anyway since you are updating them with CPU's
>DVFS). With that the clk driver won't be required to do the magic
>behind the scene.
>

That doesn't work, though, because the CPUs aren't normal devices with runtime-pm. That was the first thing I tried :).

If you think this *should* be made to work instead then I can try that.


--
Hector Martin "marcan" ([email protected])
Public key: https://mrcn.st/pub

2021-10-12 09:37:56

by Viresh Kumar

[permalink] [raw]
Subject: Re: [RFC PATCH 3/9] dt-bindings: clock: Add apple,cluster-clk binding

Apart from what Krzysztof already said:

On 12-10-21, 10:51, Krzysztof Kozlowski wrote:
> On 11/10/2021 18:57, Hector Martin wrote:
> > + pcluster_opp: opp-table-1 {
> > + compatible = "operating-points-v2";
> > + opp-shared;
> > +
> > + opp01 {
> > + opp-hz = /bits/ 64 <600000000>;
> > + opp-microvolt = <781000>;
> > + opp-level = <1>;

The opp-level thing wasn't designed to work this way, though it may
work just fine. It was designed as a unique key for power-domains,
which don't have opp-hz. The OPP core currently looks at 3 different
values, which can act as a unique key to identify the OPP. clk-rate,
bandwidth and level.

I think this is the first platform which has both hz and level in the
CPUs OPP table. What exactly is level in this case ?

Again, it may work fine, I just don't know where it may end up
breaking :)

--
viresh

2021-10-12 13:53:10

by Hector Martin

[permalink] [raw]
Subject: Re: [RFC PATCH 3/9] dt-bindings: clock: Add apple,cluster-clk binding

On 12/10/2021 18.57, Viresh Kumar wrote:
> I didn't realize earlier that we have moved out of lists :)

Whoops, sorry, I was on mobile and must've hit the wrong reply button!
My apologies.

> On 12-10-21, 18:54, Hector Martin "marcan" wrote:
>> Typically cpufreq-dt is used with clock drivers that directly take
>> the clock frequency and do whatever voodoo is necessary to set it
>> for the CPU. But here, the hardware just wants to know the index,
>> and does everything itself. So we need to encode that somewhere, to
>> avoid hardcoding it in the clock driver.
>>
>> In general, based on how these SoCs are designed, we're trying to
>> avoid having tables of volatile information in the drivers, and
>> instead keep everything in the DT. This means we have a good chance
>> that these drivers will continue to work with future SoC
>> generations, since Apple doesn't change register definitions
>> randomly most of the time.
>
> Yeah I get that and it is actually better this way. I just wanted to
> point out that we didn't think of it this way earlier :)

Yeah, makes sense. Seems to work fine :)


--
Hector Martin ([email protected])
Public Key: https://mrcn.st/pub

2021-10-14 06:54:54

by Hector Martin

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On 12/10/2021 18.32, Viresh Kumar wrote:
> On 12-10-21, 18:31, Hector Martin "marcan" wrote:
>> That doesn't work, though, because the CPUs aren't normal devices
>> with runtime-pm. That was the first thing I tried :).
>
> What's the exact problem with runtime PM here ?

The CPU devices aren't attached to their genpd, so the required OPP
transition fails with the same error.

However, this was easier to fix than I expected. With this patch to
cpufreq-dt, it all works properly, and I can drop the parent genpd
from the clock node and related handling. Thoughts?

commit c4f88743374c1f4678ee7f17fb6cae30ded9ed59
Author: Hector Martin <[email protected]>
Date: Thu Oct 14 15:47:45 2021 +0900

cpufreq: dt: Attach CPU devices to power domains

This allows the required-opps mechanism to work for CPU OPP tables,
triggering specific OPP levels in a parent power domain.

Signed-off-by: Hector Martin <[email protected]>

diff --git a/drivers/cpufreq/cpufreq-dt.c b/drivers/cpufreq/cpufreq-dt.c
index 8fcaba541539..5b22846b557d 100644
--- a/drivers/cpufreq/cpufreq-dt.c
+++ b/drivers/cpufreq/cpufreq-dt.c
@@ -16,6 +16,7 @@
#include <linux/list.h>
#include <linux/module.h>
#include <linux/of.h>
+#include <linux/pm_domain.h>
#include <linux/pm_opp.h>
#include <linux/platform_device.h>
#include <linux/regulator/consumer.h>
@@ -264,6 +265,16 @@ static int dt_cpufreq_early_init(struct device *dev, int cpu)
goto out;
}

+ /*
+ * Attach the CPU device to its genpd domain (if any), to allow OPP
+ * dependencies to be satisfied.
+ */
+ ret = genpd_dev_pm_attach(cpu_dev);
+ if (ret <= 0) {
+ dev_err(cpu_dev, "Failed to attach CPU device to genpd\n");
+ goto out;
+ }
+
/*
* The OPP table must be initialized, statically or dynamically, by this
* point.

--
Hector Martin ([email protected])
Public Key: https://mrcn.st/pub

2021-10-14 06:58:08

by Viresh Kumar

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On 14-10-21, 15:52, Hector Martin wrote:
> The CPU devices aren't attached to their genpd, so the required OPP
> transition fails with the same error.
>
> However, this was easier to fix than I expected. With this patch to
> cpufreq-dt, it all works properly, and I can drop the parent genpd
> from the clock node and related handling. Thoughts?
>
> commit c4f88743374c1f4678ee7f17fb6cae30ded9ed59
> Author: Hector Martin <[email protected]>
> Date: Thu Oct 14 15:47:45 2021 +0900
>
> cpufreq: dt: Attach CPU devices to power domains
> This allows the required-opps mechanism to work for CPU OPP tables,
> triggering specific OPP levels in a parent power domain.
> Signed-off-by: Hector Martin <[email protected]>
>
> diff --git a/drivers/cpufreq/cpufreq-dt.c b/drivers/cpufreq/cpufreq-dt.c
> index 8fcaba541539..5b22846b557d 100644
> --- a/drivers/cpufreq/cpufreq-dt.c
> +++ b/drivers/cpufreq/cpufreq-dt.c
> @@ -16,6 +16,7 @@
> #include <linux/list.h>
> #include <linux/module.h>
> #include <linux/of.h>
> +#include <linux/pm_domain.h>
> #include <linux/pm_opp.h>
> #include <linux/platform_device.h>
> #include <linux/regulator/consumer.h>
> @@ -264,6 +265,16 @@ static int dt_cpufreq_early_init(struct device *dev, int cpu)
> goto out;
> }
> + /*
> + * Attach the CPU device to its genpd domain (if any), to allow OPP
> + * dependencies to be satisfied.
> + */
> + ret = genpd_dev_pm_attach(cpu_dev);
> + if (ret <= 0) {
> + dev_err(cpu_dev, "Failed to attach CPU device to genpd\n");
> + goto out;
> + }
> +

Other platform do this from some other place I think.

Ulf, where should this code be moved ? cpu-clk driver ?

--
viresh

2021-10-14 07:06:52

by Hector Martin

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On 14/10/2021 15.56, Viresh Kumar wrote:
>> + /*
>> + * Attach the CPU device to its genpd domain (if any), to allow OPP
>> + * dependencies to be satisfied.
>> + */
>> + ret = genpd_dev_pm_attach(cpu_dev);
>> + if (ret <= 0) {
>> + dev_err(cpu_dev, "Failed to attach CPU device to genpd\n");
>> + goto out;
>> + }
>> +
>
> Other platform do this from some other place I think.
>
> Ulf, where should this code be moved ? cpu-clk driver ?
>

I see one driver that does this is drivers/clk/qcom/apcs-sdx55.c (via
dev_pm_domain_attach). Though it only does it for CPU#0; we need to do
it for all CPUs.

--
Hector Martin ([email protected])
Public Key: https://mrcn.st/pub

2021-10-14 07:24:10

by Viresh Kumar

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On 14-10-21, 16:03, Hector Martin wrote:
> On 14/10/2021 15.56, Viresh Kumar wrote:
> > > + /*
> > > + * Attach the CPU device to its genpd domain (if any), to allow OPP
> > > + * dependencies to be satisfied.
> > > + */
> > > + ret = genpd_dev_pm_attach(cpu_dev);
> > > + if (ret <= 0) {
> > > + dev_err(cpu_dev, "Failed to attach CPU device to genpd\n");
> > > + goto out;
> > > + }
> > > +
> >
> > Other platform do this from some other place I think.
> >
> > Ulf, where should this code be moved ? cpu-clk driver ?
> >
>
> I see one driver that does this is drivers/clk/qcom/apcs-sdx55.c (via
> dev_pm_domain_attach).

That may be a good place since you are already adding it and it is related to
CPU clk.

> Though it only does it for CPU#0; we need to do it
> for all CPUs.

Sure.

--
viresh

2021-10-14 07:26:29

by Hector Martin

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On 14/10/2021 16.03, Hector Martin wrote:
> On 14/10/2021 15.56, Viresh Kumar wrote:
>>> + /*
>>> + * Attach the CPU device to its genpd domain (if any), to allow OPP
>>> + * dependencies to be satisfied.
>>> + */
>>> + ret = genpd_dev_pm_attach(cpu_dev);
>>> + if (ret <= 0) {
>>> + dev_err(cpu_dev, "Failed to attach CPU device to genpd\n");
>>> + goto out;
>>> + }
>>> +
>>
>> Other platform do this from some other place I think.
>>
>> Ulf, where should this code be moved ? cpu-clk driver ?
>>
>
> I see one driver that does this is drivers/clk/qcom/apcs-sdx55.c (via
> dev_pm_domain_attach). Though it only does it for CPU#0; we need to do
> it for all CPUs.

Looking into this further, I'm not sure I like the idea of doing this in
the clocks driver. There might be locking issues since it gets
instantiated twice and yet doesn't really itself know what subset of
CPUs it applies to.

There's another driver that does this:
drivers/cpuidle/cpuidle-psci-domain.c. That one specifically looks for a
power domain called "psci". Perhaps it would make sense to make this
generic in cpufreq-dt as per my prior patch, but explicitly request a
"cpufreq" domain? That way only devicetrees that opt in to having this
handled by cpufreq by naming it that way would get this behavior.

--
Hector Martin ([email protected])
Public Key: https://mrcn.st/pub

2021-10-14 09:57:48

by Ulf Hansson

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On Tue, 12 Oct 2021 at 07:57, Hector Martin <[email protected]> wrote:
>
> On 12/10/2021 14.51, Viresh Kumar wrote:
> > On 12-10-21, 14:34, Hector Martin wrote:
> >> The table *is* assigned to a genpd (the memory controller), it's just that
> >> that genpd isn't actually a parent of the CPU device. Without the patch you
> >> end up with:
> >>
> >> [ 3.040060] cpu cpu4: Failed to set performance rate of cpu4: 0 (-19)
> >> [ 3.042881] cpu cpu4: Failed to set required opps: -19
> >> [ 3.045508] cpufreq: __target_index: Failed to change cpu frequency: -19
> >
> > Hmm, Saravana and Sibi were working on a similar problem earlier and decided to
> > solve this using devfreq instead. Don't remember the exact series which got
> > merged for this, Sibi ?
> >
> > If this part fails, how do you actually set the performance state of the memory
> > controller's genpd ?
>
> The clock controller has the genpd as an actual power-domain parent, so
> it does it instead. From patch #7:
>
> > + if (cluster->has_pd)
> > + dev_pm_genpd_set_performance_state(cluster->dev,
> > + dev_pm_opp_get_required_pstate(opp, 0));
> > +
>
> This is arguably not entirely representative of how the hardware works,
> since technically the cluster switching couldn't care less what the
> memory controller is doing; it's a soft dependency, states that should
> be switched together but are not interdependent (in fact, the clock code
> does this unconditionally after the CPU p-state change, regardless of
> whether we're shifting up or down; this is, FWIW, the same order macOS
> uses, and it clearly doesn't matter which way you do it).

Yes, this sounds like you should move away from modeling the memory
part as a parent genpd for the CPUs' genpd.

As Viresh pointed out, a devfreq driver seems like a better way to do
this. As a matter of fact, there are already devfreq drivers that do
this, unless I am mistaken.

It looks like devfreq providers are listening to opp/cpufreq
notifiers, as to get an indication of when it could make sense to
change a performance state.

In some cases the devfreq provider is also modeled as an interconnect
provider, allowing consumers to specify memory bandwidth constraints,
which may trigger a new performance state to be set for the memory
controller.

In the tegra case, the memory controller is modelled as an
interconnect provider and the devfreq node is modelled as an
interconnect-consumer of the memory controller. Perhaps this can work
for apple SoCs too?

That said, perhaps as an option to move forward, we can try to get the
cpufreq pieces solved first. Then as a step on top, add the
performance scaling for the memory controller?

>
> --
> Hector Martin ([email protected])
> Public Key: https://mrcn.st/pub

Kind regards
Uffe

2021-10-14 11:12:18

by Ulf Hansson

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On Thu, 14 Oct 2021 at 09:23, Hector Martin <[email protected]> wrote:
>
> On 14/10/2021 16.03, Hector Martin wrote:
> > On 14/10/2021 15.56, Viresh Kumar wrote:
> >>> + /*
> >>> + * Attach the CPU device to its genpd domain (if any), to allow OPP
> >>> + * dependencies to be satisfied.
> >>> + */
> >>> + ret = genpd_dev_pm_attach(cpu_dev);
> >>> + if (ret <= 0) {
> >>> + dev_err(cpu_dev, "Failed to attach CPU device to genpd\n");
> >>> + goto out;
> >>> + }
> >>> +
> >>
> >> Other platform do this from some other place I think.
> >>
> >> Ulf, where should this code be moved ? cpu-clk driver ?
> >>
> >
> > I see one driver that does this is drivers/clk/qcom/apcs-sdx55.c (via
> > dev_pm_domain_attach). Though it only does it for CPU#0; we need to do
> > it for all CPUs.
>
> Looking into this further, I'm not sure I like the idea of doing this in
> the clocks driver. There might be locking issues since it gets
> instantiated twice and yet doesn't really itself know what subset of
> CPUs it applies to.

I agree. I suggest you look into using a genpd provider and hook up
all CPU's devices to it. I think that is what Viresh also suggested
earlier - and this makes most sense to me.

As a reference you may have a look at some Qcom platforms that already use this:

arch/arm64/boot/dts/qcom/qcs404.dtsi

drivers/cpufreq/qcom-cpufreq-nvmem.c:
To hook up CPU devices to their PM domains (genpds) - it calls
dev_pm_opp_attach_genpd(), which is a kind of wrapper for
dev_pm_domain_attach_by_name().

drivers/soc/qcom/cpr.c
Registers the genpd provider that is capable of dealing with
performance states/OPPs for CPUs.

>
> There's another driver that does this:
> drivers/cpuidle/cpuidle-psci-domain.c. That one specifically looks for a
> power domain called "psci". Perhaps it would make sense to make this
> generic in cpufreq-dt as per my prior patch, but explicitly request a
> "cpufreq" domain? That way only devicetrees that opt in to having this
> handled by cpufreq by naming it that way would get this behavior.

That sounds like an idea that is worth exploring. In this way, the
only thing that needs to be implemented for new cases would be the
genpd provider driver.

BTW, as you will figure out by looking at the above references, for
the qcom case we are using "cpr" as the domain name for cpufreq. Of
course, that doesn't mean we can use "cpufreq" (or whatever name that
makes sense) going forward for new cases.

>
> --
> Hector Martin ([email protected])
> Public Key: https://mrcn.st/pub

Kind regards
Uffe

2021-10-14 13:07:34

by Hector Martin

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On 14/10/2021 18.55, Ulf Hansson wrote:
> Yes, this sounds like you should move away from modeling the memory
> part as a parent genpd for the CPUs' genpd.
>
> As Viresh pointed out, a devfreq driver seems like a better way to do
> this. As a matter of fact, there are already devfreq drivers that do
> this, unless I am mistaken.
>
> It looks like devfreq providers are listening to opp/cpufreq
> notifiers, as to get an indication of when it could make sense to
> change a performance state.
>
> In some cases the devfreq provider is also modeled as an interconnect
> provider, allowing consumers to specify memory bandwidth constraints,
> which may trigger a new performance state to be set for the memory
> controller.
>
> In the tegra case, the memory controller is modelled as an
> interconnect provider and the devfreq node is modelled as an
> interconnect-consumer of the memory controller. Perhaps this can work
> for apple SoCs too?

I was poking around and noticed the OPP core can already integrate with
interconnect requirements, so perhaps the memory controller can be an
interconnect provider, and the CPU nodes can directly reference it as a
consumer? This seems like a more accurate model of what the hardware
does, and I think I saw some devices doing this already.

(only problem is I have no idea of the actual bandwidth numbers involved
here... I'll have to run some benchmarks to make sure this isn't just
completely dummy data)

>
> That said, perhaps as an option to move forward, we can try to get the
> cpufreq pieces solved first. Then as a step on top, add the
> performance scaling for the memory controller?

Sure; that's a pretty much independent part of this patchset, though I'm
thinking I might as well try some things out for v2 anyway; if it looks
like it'll take longer we can split it out and do just the cpufreq side.

--
Hector Martin ([email protected])
Public Key: https://mrcn.st/pub

2021-10-14 14:56:23

by Ulf Hansson

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On Thu, 14 Oct 2021 at 13:43, Hector Martin <[email protected]> wrote:
>
> On 14/10/2021 18.55, Ulf Hansson wrote:
> > Yes, this sounds like you should move away from modeling the memory
> > part as a parent genpd for the CPUs' genpd.
> >
> > As Viresh pointed out, a devfreq driver seems like a better way to do
> > this. As a matter of fact, there are already devfreq drivers that do
> > this, unless I am mistaken.
> >
> > It looks like devfreq providers are listening to opp/cpufreq
> > notifiers, as to get an indication of when it could make sense to
> > change a performance state.
> >
> > In some cases the devfreq provider is also modeled as an interconnect
> > provider, allowing consumers to specify memory bandwidth constraints,
> > which may trigger a new performance state to be set for the memory
> > controller.
> >
> > In the tegra case, the memory controller is modelled as an
> > interconnect provider and the devfreq node is modelled as an
> > interconnect-consumer of the memory controller. Perhaps this can work
> > for apple SoCs too?
>
> I was poking around and noticed the OPP core can already integrate with
> interconnect requirements, so perhaps the memory controller can be an
> interconnect provider, and the CPU nodes can directly reference it as a
> consumer? This seems like a more accurate model of what the hardware
> does, and I think I saw some devices doing this already.

Yeah, that could work too. And, yes, I agree, it may be a better
description of the HW.

>
> (only problem is I have no idea of the actual bandwidth numbers involved
> here... I'll have to run some benchmarks to make sure this isn't just
> completely dummy data)
>
> >
> > That said, perhaps as an option to move forward, we can try to get the
> > cpufreq pieces solved first. Then as a step on top, add the
> > performance scaling for the memory controller?
>
> Sure; that's a pretty much independent part of this patchset, though I'm
> thinking I might as well try some things out for v2 anyway; if it looks
> like it'll take longer we can split it out and do just the cpufreq side.

In any case, I do my best to help with review.

Kind regards
Uffe

2021-10-14 21:36:54

by Hector Martin

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On 14/10/2021 21.55, Ulf Hansson wrote:
> On Thu, 14 Oct 2021 at 13:43, Hector Martin <[email protected]> wrote:
>> I was poking around and noticed the OPP core can already integrate with
>> interconnect requirements, so perhaps the memory controller can be an
>> interconnect provider, and the CPU nodes can directly reference it as a
>> consumer? This seems like a more accurate model of what the hardware
>> does, and I think I saw some devices doing this already.
>
> Yeah, that could work too. And, yes, I agree, it may be a better
> description of the HW.
>
>>
>> (only problem is I have no idea of the actual bandwidth numbers involved
>> here... I'll have to run some benchmarks to make sure this isn't just
>> completely dummy data)
>>

So... I tried getting bandwidth numbers and failed. It seems these
registers don't actually affect peak performance in any measurable way.
I'm also getting almost the same GeekBench scores on macOS with and
without this mechanism enabled, although there is one subtest that seems
to show a measurable difference.

My current guess is this is something more subtle (latencies? idle
timers and such?) than a performance state. If that is the case, do you
have any ideas as to the best way to model it in Linux? Should we even
bother if it mostly has a minimal performance gain for typical workloads?

I'll try to do some latency tests, see if I can make sense of what it's
actually doing.

--
Hector Martin ([email protected])
Public Key: https://mrcn.st/pub

2021-10-15 20:58:23

by Ulf Hansson

[permalink] [raw]
Subject: Re: [RFC PATCH 4/9] opp: core: Don't warn if required OPP device does not exist

On Thu, 14 Oct 2021 at 19:02, Hector Martin <[email protected]> wrote:
>
> On 14/10/2021 21.55, Ulf Hansson wrote:
> > On Thu, 14 Oct 2021 at 13:43, Hector Martin <[email protected]> wrote:
> >> I was poking around and noticed the OPP core can already integrate with
> >> interconnect requirements, so perhaps the memory controller can be an
> >> interconnect provider, and the CPU nodes can directly reference it as a
> >> consumer? This seems like a more accurate model of what the hardware
> >> does, and I think I saw some devices doing this already.
> >
> > Yeah, that could work too. And, yes, I agree, it may be a better
> > description of the HW.
> >
> >>
> >> (only problem is I have no idea of the actual bandwidth numbers involved
> >> here... I'll have to run some benchmarks to make sure this isn't just
> >> completely dummy data)
> >>
>
> So... I tried getting bandwidth numbers and failed. It seems these
> registers don't actually affect peak performance in any measurable way.
> I'm also getting almost the same GeekBench scores on macOS with and
> without this mechanism enabled, although there is one subtest that seems
> to show a measurable difference.
>
> My current guess is this is something more subtle (latencies? idle
> timers and such?) than a performance state. If that is the case, do you
> have any ideas as to the best way to model it in Linux? Should we even
> bother if it mostly has a minimal performance gain for typical workloads?

For latency constraints, we have dev_pm_qos. This will make the genpd
governor, to prevent deeper idle states for the device and its
corresponding PM domain (genpd). But that doesn't sound like a good
fit here.

If you are right, it rather sounds like there is some kind of
quiescence mode of the memory controller that can be prevented. But I
have no clue, of course. :-)

>
> I'll try to do some latency tests, see if I can make sense of what it's
> actually doing.
>

Kind regards
Uffe

2021-10-19 22:47:35

by Rob Herring (Arm)

[permalink] [raw]
Subject: Re: [RFC PATCH 2/9] dt-bindings: memory-controller: Add apple,mcc binding

On Tue, Oct 12, 2021 at 10:48:12AM +0200, Krzysztof Kozlowski wrote:
> On 11/10/2021 18:57, Hector Martin wrote:
> > This device represents the memory controller in Apple SoCs, and is
> > chiefly in charge of adjusting performance characteristics according to
> > system demand.
> >
> > Signed-off-by: Hector Martin <[email protected]>
> > ---
> > .../memory-controllers/apple,mcc.yaml | 80 +++++++++++++++++++
> > .../opp/apple,mcc-operating-points.yaml | 62 ++++++++++++++
> > 2 files changed, 142 insertions(+)
> > create mode 100644 Documentation/devicetree/bindings/memory-controllers/apple,mcc.yaml
> > create mode 100644 Documentation/devicetree/bindings/opp/apple,mcc-operating-points.yaml
> >
> > diff --git a/Documentation/devicetree/bindings/memory-controllers/apple,mcc.yaml b/Documentation/devicetree/bindings/memory-controllers/apple,mcc.yaml
> > new file mode 100644
> > index 000000000000..0774f10e65ed
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/memory-controllers/apple,mcc.yaml
> > @@ -0,0 +1,80 @@
> > +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> > +%YAML 1.2
> > +---
> > +$id: http://devicetree.org/schemas/memory-controllers/apple,mcc.yaml#
> > +$schema: http://devicetree.org/meta-schemas/core.yaml#
> > +
> > +title: Apple SoC MCC memory controller performance controls
> > +
> > +maintainers:
> > + - Hector Martin <[email protected]>
> > +
> > +description: |
> > + Apple SoCs contain a multichannel memory controller that can have its
> > + configuration changed to adjust to changing performance requirements from
> > + the rest of the SoC. This node represents the controller and provides a
> > + power domain provider that downstream devices can use to adjust the memory
> > + controller performance level.
> > +
> > +properties:
> > + compatible:
> > + items:
> > + - enum:
> > + - apple,t8103-mcc
> > + - const: apple,mcc
> > +
> > + reg:
> > + maxItems: 1
> > +
> > + "#power-domain-cells":
> > + const: 0
> > +
> > + operating-points-v2:
> > + $ref: /schemas/types.yaml#/definitions/phandle-array
> > + description:
> > + A reference to the OPP table describing the memory controller performance
> > + levels. Each OPP node should contain an `apple,memory-perf-config`
> > + property that contains the configuration values for that performance
> > + level.
> > +
> > + apple,num-channels:
> > + $ref: /schemas/types.yaml#/definitions/uint32
> > + description:
> > + The number of memory channels in use.
> > +
> > +required:
> > + - compatible
> > + - reg
> > + - "#power-domain-cells"
> > + - operating-points-v2
> > + - apple,num-channels
> > +
> > +additionalProperties: false
> > +
> > +examples:
> > + # See clock/apple,cluster-clock.yaml for an example of downstream usage.
> > + - |
> > + mcc_opp: opp-table-2 {
> > + compatible = "operating-points-v2";
>
> apple,mcc-operating-points?

+1


> > +
> > + mcc_lowperf: opp0 {
> > + opp-level = <0>;
> > + apple,memory-perf-config = <0x813057f 0x1800180>;
> > + };
> > + mcc_highperf: opp1 {
> > + opp-level = <1>;
> > + apple,memory-perf-config = <0x133 0x55555340>;
> > + };
> > + };
> > + soc {
> > + #address-cells = <2>;
> > + #size-cells = <2>;
> > +
> > + mcc: memory-controller@200200000 {
> > + compatible = "apple,t8103-mcc", "apple,mcc";
> > + #power-domain-cells = <0>;
> > + reg = <0x2 0x200000 0x0 0x200000>;
> > + operating-points-v2 = <&mcc_opp>;
> > + apple,num-channels = <8>;
> > + };
> > + };
> > diff --git a/Documentation/devicetree/bindings/opp/apple,mcc-operating-points.yaml b/Documentation/devicetree/bindings/opp/apple,mcc-operating-points.yaml
> > new file mode 100644
> > index 000000000000..babf27841bb7
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/opp/apple,mcc-operating-points.yaml
> > @@ -0,0 +1,62 @@
> > +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> > +%YAML 1.2
> > +---
> > +$id: http://devicetree.org/schemas/opp/apple,mcc-operating-points.yaml#
> > +$schema: http://devicetree.org/meta-schemas/core.yaml#
> > +
> > +title: Apple SoC memory controller OPP bindings
> > +
> > +maintainers:
> > + - Hector Martin <[email protected]>
> > +
> > +description: |
> > + Apple SoCs can have their memory controller performance adjusted depending on
> > + system requirements. These performance states are represented by specific
> > + memory controller register values. The apple-mcc driver uses these values
> > + to change the MCC performance.
> > +
> > +allOf:
> > + - $ref: opp-v2-base.yaml#
> > +
> > +properties:
> > + compatible:
> > + const: apple,mcc-operating-points
> > +
> > +required:
> > + - compatible
> > +
> > +patternProperties:
> > + "opp[0-9]+":
> > + type: object
> > +
> > + properties:
> > + opp-level: true
>
> You don't need to mention it.

Actually, you do.

You are thinking unevaluatedProperties takes care of it, but it doesn't
here. The problem is if you have 2 schemas (this one and
opp-v2-base.yaml) with child nodes, the child nodes in each schema are
evaluated separately.

So anywhere we have child nodes, we need the child node schema to be a
separate file or able to be directly referenced (i.e. under $defs). I
only realized this when testing out unevaluatedProperties support.

Rob