Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <6407af2a-18c6-9baf-cc9b-dcf7001812b7@linaro.org>
Date:   Wed, 19 Apr 2023 13:31:01 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.9.1
Subject: Re: [PATCH RFT v2 01/14] dt-bindings: clock: qcom,rpmcc: Add a way to
 enable unused clock cleanup
Content-Language: en-US
From:   Konrad Dybcio <konrad.dybcio@linaro.org>
To:     Stephen Boyd <sboyd@kernel.org>,
        Stephan Gerhold <stephan@gerhold.net>
Cc:     Andy Gross <agross@kernel.org>,
        Bjorn Andersson <andersson@kernel.org>,
        Michael Turquette <mturquette@baylibre.com>,
        Rob Herring <robh+dt@kernel.org>,
        Krzysztof Kozlowski <krzysztof.kozlowski+dt@linaro.org>,
        linux-arm-msm@vger.kernel.org, linux-clk@vger.kernel.org,
        linux-kernel@vger.kernel.org,
        Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>,
        devicetree@vger.kernel.org
References: <20230303-topic-rpmcc_sleep-v2-0-ae80a325fe94@linaro.org>
 <20230303-topic-rpmcc_sleep-v2-1-ae80a325fe94@linaro.org>
 <ZD2YYrOdQMD3pi7u@gerhold.net>
 <d63d4896afe8a1a901470f88862ce608.sboyd@kernel.org>
 <3873483f-7f7d-a146-cca9-b50f054289d4@linaro.org>
In-Reply-To: <3873483f-7f7d-a146-cca9-b50f054289d4@linaro.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 18.04.2023 12:33, Konrad Dybcio wrote:
> 
> 
> On 18.04.2023 02:19, Stephen Boyd wrote:
>> Quoting Stephan Gerhold (2023-04-17 12:05:06)
>>> On Wed, Mar 08, 2023 at 10:35:17PM +0100, Konrad Dybcio wrote:
>>>> Disabling RPMCC clocks can be a bit touchy. If we can't guarantee all
>>>> (or at least most) of the oneline peripherals ask the interconnect
>>>> framework to keep their buses online and guarantee enough bandwidth,
>>>> we're relying on bootloader defaults to keep the said buses alive through
>>>> RPM requests and rate setting on RPM clocks.
>>>>
>>>> Without that in place, the RPM clocks are never enabled in the CCF, which
>>>> qualifies them to be cleaned up, since - as far as Linux is concerned -
>>>> nobody's using them and they're just wasting power. Doing so will end
>>>> tragically, as within miliseconds we'll get *some* access attempt on an
>>>> unlocked bus which will cause a platform crash.
>>>>
>>>> On the other hand, if we want to save power and put well-supported
>>>> platforms to sleep, we should be shutting off at least some of these
>>>> clocks (this time with a clear distinction of which ones are *actually*
>>>> not in use, coming from the interconnect driver).
>>>>
>>>> To differentiate between these two cases while not breaking older DTs,
>>>> introduce an opt-in property to correctly mark RPM clocks as enabled
>>>> after handoff (the initial max freq vote) and hence qualify them for the
>>>> common unused clock cleanup.
>>>>
>>>> Signed-off-by: Konrad Dybcio <konrad.dybcio@linaro.org>
>>>> ---
>>>>  Documentation/devicetree/bindings/clock/qcom,rpmcc.yaml | 6 ++++++
>>>>  1 file changed, 6 insertions(+)
>>>>
>>>> diff --git a/Documentation/devicetree/bindings/clock/qcom,rpmcc.yaml b/Documentation/devicetree/bindings/clock/qcom,rpmcc.yaml
>>>> index 2a95bf8664f9..386153f61971 100644
>>>> --- a/Documentation/devicetree/bindings/clock/qcom,rpmcc.yaml
>>>> +++ b/Documentation/devicetree/bindings/clock/qcom,rpmcc.yaml
>>>> @@ -58,6 +58,12 @@ properties:
>>>>      minItems: 1
>>>>      maxItems: 2
>>>>  
>>>> +  qcom,clk-disable-unused:
>>>> +    type: boolean
>>>> +    description:
>>>> +      Indicates whether unused RPM clocks can be shut down with the common
>>>> +      unused clock cleanup. Requires a functional interconnect driver.
>>>> +
>>>
>>> I'm surprised that Stephen Boyd did not bring up his usual "rant" here
>>> of moving the interconnect clock voting out of rpmcc into the
>>> interconnect drivers (see [1], [2]). :-)
>>
>> :) I was hoping to get a fix for disabling unused clks during late init
>> at the same time. Shucks!
>>
>>>
>>> I was a bit "cautious" about it back then but at this point I think it
>>> kind of makes sense. Make sure to read Stephen's detailed explanation in
>>> https://lore.kernel.org/linux-arm-msm/159796605593.334488.8355244657387381953@swboyd.mtv.corp.google.com/
>>>
>>> We keep looking for workarounds to prevent the CCF from "messing" with
>>> interconnect-related clocks. But the CCF cannot mess with "clocks" it
>>> does not manage. The RPM interconnect drivers already talk directly to
>>> the RPM in drivers/interconnect/qcom/smd-rpm.c. I think it should be
>>> quite easy to move the QCOM_SMD_RPM_BUS_CLK relates defines over there
>>> and just bypass the CCF entirely.
>>
>> Please do it!
> Okay, that's a plan..
> 
>>
>>>
>>> For backwards compatibility (for platforms without interconnect drivers)
>>> one could either assume that the bootloader bandwidth votes will be
>>> sufficient and just leave those clocks completely alone. Or the
>>> "icc_smd_rpm" platform device could initially make max votes similar to
>>> the rpmcc device. By coincidence the "icc_smd_rpm" platform device is
>>> always created, no matter how the device tree looks or if the platform
>>> actually has an interconnect driver.
>>>
>>
>> Yeah that's a good plan. Suspend will be broken or burn a lot of power,
> (that's what happens as of today, so sgtm!)
> 
>> but presumably the new DTB will be used fairly quickly. Or you can
>> implement something like clkdev for interconnects that lets you hack up
>> an association between interconnects and consumers for existing DTs and
>> then drop those lookups months later.
> Uh.. let's not.. Let's just contain it in the interconnect driver.
> 
> The buses will be at bearable frequencies coming from the bootloader
> (as RPM, storage etc. are enabled) and boosting them at icc_rpm_smd
> probe sounds sane.
What should we do about the non-bus RPM clocks though? I don't fancy
IPA_CLK running 24/7.. And Stephan Gerhold was able to achieve VDD_MIN
on msm8909 with these clocks shut down (albeit with a very basic dt setup)!

Taking into account the old interconnect-enabled DTs, some of the
clocks would need to be on so that the QoS writes can succeed
(e.g. the MAS_IPA endpoint needs IPA_CLK), it gets complicated again..

I suppose something like this would work-ish:

0. remove clock handles as they're now contained within icc and
   use them as a "legacy marker"
1. add:
	if (qp->bus_clocks)
		// skip qos writes

This will:
- let us add is_enabled so that all RPM clocks bar XO_A will be cleaned up
- save massively on code complexity

at the cost of retroactively removing features (QoS settings) for people
with old DTs and new kernels (don't tell Torvalds!)

This DTB ABI stuff really gets in the way sometimes :/ We're only now
fixing up U-Boot to be able to use upstream Linux DTs and other than
that I think only OpenBSD uses it with 8280.. Wish we could get rid of
all old junk once and then establish immutability but oh well..

Konrad
> 
> Konrad