Devfreq framework supports 2 modes for monitoring devices.
Use delayed timer as default instead of deferrable timer
in order to monitor the GPU status regardless of CPU idle.
Signed-off-by: Lukasz Luba <[email protected]>
---
Hi all,
I've missed the Lima driver while working on Panfrost patch for fixing
the issue with default devfreq framework polling mode. More about this
and the patch, can be found here [1].
Regards,
Lukasz Luba
[1] https://lore.kernel.org/lkml/[email protected]/
drivers/gpu/drm/lima/lima_devfreq.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/gpu/drm/lima/lima_devfreq.c b/drivers/gpu/drm/lima/lima_devfreq.c
index 5686ad4aaf7c..f1c9eb3e71bd 100644
--- a/drivers/gpu/drm/lima/lima_devfreq.c
+++ b/drivers/gpu/drm/lima/lima_devfreq.c
@@ -81,6 +81,7 @@ static int lima_devfreq_get_dev_status(struct device *dev,
}
static struct devfreq_dev_profile lima_devfreq_profile = {
+ .timer = DEVFREQ_TIMER_DELAYED,
.polling_ms = 50, /* ~3 frames */
.target = lima_devfreq_target,
.get_dev_status = lima_devfreq_get_dev_status,
--
2.17.1
Thanks for the patch. But I can't observe any difference on glmark2
with or without this patch.
Maybe you can provide other test which can benefit from it.
Considering it will wake up CPU more frequently, and user may choose
to change this by sysfs,
I'd like to not apply it.
Regards,
Qiang
On Wed, Jan 27, 2021 at 6:51 PM Lukasz Luba <[email protected]> wrote:
>
> Devfreq framework supports 2 modes for monitoring devices.
> Use delayed timer as default instead of deferrable timer
> in order to monitor the GPU status regardless of CPU idle.
>
> Signed-off-by: Lukasz Luba <[email protected]>
> ---
> Hi all,
>
> I've missed the Lima driver while working on Panfrost patch for fixing
> the issue with default devfreq framework polling mode. More about this
> and the patch, can be found here [1].
>
> Regards,
> Lukasz Luba
>
> [1] https://lore.kernel.org/lkml/[email protected]/
>
> drivers/gpu/drm/lima/lima_devfreq.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/drivers/gpu/drm/lima/lima_devfreq.c b/drivers/gpu/drm/lima/lima_devfreq.c
> index 5686ad4aaf7c..f1c9eb3e71bd 100644
> --- a/drivers/gpu/drm/lima/lima_devfreq.c
> +++ b/drivers/gpu/drm/lima/lima_devfreq.c
> @@ -81,6 +81,7 @@ static int lima_devfreq_get_dev_status(struct device *dev,
> }
>
> static struct devfreq_dev_profile lima_devfreq_profile = {
> + .timer = DEVFREQ_TIMER_DELAYED,
> .polling_ms = 50, /* ~3 frames */
> .target = lima_devfreq_target,
> .get_dev_status = lima_devfreq_get_dev_status,
> --
> 2.17.1
>
Hi Qiang,
On 1/30/21 1:51 PM, Qiang Yu wrote:
> Thanks for the patch. But I can't observe any difference on glmark2
> with or without this patch.
> Maybe you can provide other test which can benefit from it.
This is a design problem and has impact on the whole system.
There is a few issues. When the device is not checked and there are
long delays between last check and current, the history is broken.
It confuses the devfreq governor and thermal governor (Intelligent Power
Allocation (IPA)). Thermal governor works on stale stats data and makes
stupid decisions, because there is no new stats (device not checked).
Similar applies to devfreq simple_ondemand governor, where it 'tires' to
work on a loooong period even 3sec and make prediction for the next
frequency based on it (which is broken).
How it should be done: constant reliable check is needed, then:
- period is guaranteed and has fixed size, e.g 50ms or 100ms.
- device status is quite recent so thermal devfreq cooling provides
'fresh' data into thermal governor
This would prevent odd behavior and solve the broken cases.
>
> Considering it will wake up CPU more frequently, and user may choose
> to change this by sysfs,
> I'd like to not apply it.
The deferred timer for GPU is wrong option, for UFS or eMMC makes more
sense. It's also not recommended for NoC busses. I've discovered that
some time ago and proposed to have option to switch into delayed timer.
Trust me, it wasn't obvious to find out that this missing check has
those impacts. So the other engineers or users might not know that some
problems they faces (especially when the device load is changing) is due
to this delayed vs deffered timer and they will change it in the sysfs.
Regards,
Lukasz
>
> Regards,
> Qiang
>
Hi Lukasz,
Thanks for the explanation. So the deferred timer option makes a mistake that
when GPU goes from idle to busy for only one poll periodic, in this
case 50ms, right?
But delayed timer will wakeup CPU every 50ms even when system is idle, will this
cause more power consumption for the case like phone suspend?
Regards,
Qiang
On Mon, Feb 1, 2021 at 5:53 PM Lukasz Luba <[email protected]> wrote:
>
> Hi Qiang,
>
> On 1/30/21 1:51 PM, Qiang Yu wrote:
> > Thanks for the patch. But I can't observe any difference on glmark2
> > with or without this patch.
> > Maybe you can provide other test which can benefit from it.
>
> This is a design problem and has impact on the whole system.
> There is a few issues. When the device is not checked and there are
> long delays between last check and current, the history is broken.
> It confuses the devfreq governor and thermal governor (Intelligent Power
> Allocation (IPA)). Thermal governor works on stale stats data and makes
> stupid decisions, because there is no new stats (device not checked).
> Similar applies to devfreq simple_ondemand governor, where it 'tires' to
> work on a loooong period even 3sec and make prediction for the next
> frequency based on it (which is broken).
>
> How it should be done: constant reliable check is needed, then:
> - period is guaranteed and has fixed size, e.g 50ms or 100ms.
> - device status is quite recent so thermal devfreq cooling provides
> 'fresh' data into thermal governor
>
> This would prevent odd behavior and solve the broken cases.
>
> >
> > Considering it will wake up CPU more frequently, and user may choose
> > to change this by sysfs,
> > I'd like to not apply it.
>
> The deferred timer for GPU is wrong option, for UFS or eMMC makes more
> sense. It's also not recommended for NoC busses. I've discovered that
> some time ago and proposed to have option to switch into delayed timer.
> Trust me, it wasn't obvious to find out that this missing check has
> those impacts. So the other engineers or users might not know that some
> problems they faces (especially when the device load is changing) is due
> to this delayed vs deffered timer and they will change it in the sysfs.
>
> Regards,
> Lukasz
>
> >
> > Regards,
> > Qiang
> >
On 2/2/21 1:01 AM, Qiang Yu wrote:
> Hi Lukasz,
>
> Thanks for the explanation. So the deferred timer option makes a mistake that
> when GPU goes from idle to busy for only one poll periodic, in this
> case 50ms, right?
Not exactly. Driver sets the polling interval to 50ms (in this case)
because it needs ~3-frame average load (in 60fps). I have discovered the
issue quite recently that on systems with 2 CPUs or more, the devfreq
core is not monitoring the devices even for seconds. Therefore, we might
end up with quite big amount of work that GPU is doing, but we don't
know about it. Devfreq core didn't check <- timer didn't fired. Then
suddenly that CPU, which had the deferred timer registered last time,
is waking up and timer triggers to check our device. We get the stats,
but they might be showing load from 1sec not 50ms. We feed them into
governor. Governor sees the new load, but was tested and configured for
50ms, so it might try to rise the frequency to max. The GPU work might
be already lower and there is no need for such freq. Then the CPU goes
idle again, so no devfreq core check for next e.g. 1sec, but the
frequency stays at max OPP and we burn power.
So, it's completely unreliable. We might stuck at min frequency and
suffer the frame drops, or sometimes stuck to max freq and burn more
power when there is no such need.
Similar for thermal governor, which is confused by this old stats and
long period stats, longer than 50ms.
Stats from last e.g. ~1sec tells you nothing about real recent GPU
workload.
> But delayed timer will wakeup CPU every 50ms even when system is idle, will this
> cause more power consumption for the case like phone suspend?
No, in case of phone suspend it won't increase the power consumption.
The device won't be woken up, it will stay in suspend.
Regards,
Lukasz
>
> Regards,
> Qiang
>
>
> On Mon, Feb 1, 2021 at 5:53 PM Lukasz Luba <[email protected]> wrote:
>>
>> Hi Qiang,
>>
>> On 1/30/21 1:51 PM, Qiang Yu wrote:
>>> Thanks for the patch. But I can't observe any difference on glmark2
>>> with or without this patch.
>>> Maybe you can provide other test which can benefit from it.
>>
>> This is a design problem and has impact on the whole system.
>> There is a few issues. When the device is not checked and there are
>> long delays between last check and current, the history is broken.
>> It confuses the devfreq governor and thermal governor (Intelligent Power
>> Allocation (IPA)). Thermal governor works on stale stats data and makes
>> stupid decisions, because there is no new stats (device not checked).
>> Similar applies to devfreq simple_ondemand governor, where it 'tires' to
>> work on a loooong period even 3sec and make prediction for the next
>> frequency based on it (which is broken).
>>
>> How it should be done: constant reliable check is needed, then:
>> - period is guaranteed and has fixed size, e.g 50ms or 100ms.
>> - device status is quite recent so thermal devfreq cooling provides
>> 'fresh' data into thermal governor
>>
>> This would prevent odd behavior and solve the broken cases.
>>
>>>
>>> Considering it will wake up CPU more frequently, and user may choose
>>> to change this by sysfs,
>>> I'd like to not apply it.
>>
>> The deferred timer for GPU is wrong option, for UFS or eMMC makes more
>> sense. It's also not recommended for NoC busses. I've discovered that
>> some time ago and proposed to have option to switch into delayed timer.
>> Trust me, it wasn't obvious to find out that this missing check has
>> those impacts. So the other engineers or users might not know that some
>> problems they faces (especially when the device load is changing) is due
>> to this delayed vs deffered timer and they will change it in the sysfs.
>>
>> Regards,
>> Lukasz
>>
>>>
>>> Regards,
>>> Qiang
>>>
On Tue, Feb 2, 2021 at 10:02 PM Lukasz Luba <[email protected]> wrote:
>
>
>
> On 2/2/21 1:01 AM, Qiang Yu wrote:
> > Hi Lukasz,
> >
> > Thanks for the explanation. So the deferred timer option makes a mistake that
> > when GPU goes from idle to busy for only one poll periodic, in this
> > case 50ms, right?
>
> Not exactly. Driver sets the polling interval to 50ms (in this case)
> because it needs ~3-frame average load (in 60fps). I have discovered the
> issue quite recently that on systems with 2 CPUs or more, the devfreq
> core is not monitoring the devices even for seconds. Therefore, we might
> end up with quite big amount of work that GPU is doing, but we don't
> know about it. Devfreq core didn't check <- timer didn't fired. Then
> suddenly that CPU, which had the deferred timer registered last time,
> is waking up and timer triggers to check our device. We get the stats,
> but they might be showing load from 1sec not 50ms. We feed them into
> governor. Governor sees the new load, but was tested and configured for
> 50ms, so it might try to rise the frequency to max. The GPU work might
> be already lower and there is no need for such freq. Then the CPU goes
> idle again, so no devfreq core check for next e.g. 1sec, but the
> frequency stays at max OPP and we burn power.
>
> So, it's completely unreliable. We might stuck at min frequency and
> suffer the frame drops, or sometimes stuck to max freq and burn more
> power when there is no such need.
>
> Similar for thermal governor, which is confused by this old stats and
> long period stats, longer than 50ms.
>
> Stats from last e.g. ~1sec tells you nothing about real recent GPU
> workload.
Oh, right, I missed this case.
>
> > But delayed timer will wakeup CPU every 50ms even when system is idle, will this
> > cause more power consumption for the case like phone suspend?
>
> No, in case of phone suspend it won't increase the power consumption.
> The device won't be woken up, it will stay in suspend.
I mean the CPU is waked up frequently by timer when phone suspend,
not the whole device (like the display).
Seems it's better to have deferred timer when device is suspended for
power saving,
and delayed timer when device in working state. User knows this and
can use sysfs
to change it.
Set the delayed timer as default is reasonable, so patch is:
Reviewed-by: Qiang Yu <[email protected]>
Regards,
Qiang
>
> Regards,
> Lukasz
>
>
> >
> > Regards,
> > Qiang
> >
> >
> > On Mon, Feb 1, 2021 at 5:53 PM Lukasz Luba <[email protected]> wrote:
> >>
> >> Hi Qiang,
> >>
> >> On 1/30/21 1:51 PM, Qiang Yu wrote:
> >>> Thanks for the patch. But I can't observe any difference on glmark2
> >>> with or without this patch.
> >>> Maybe you can provide other test which can benefit from it.
> >>
> >> This is a design problem and has impact on the whole system.
> >> There is a few issues. When the device is not checked and there are
> >> long delays between last check and current, the history is broken.
> >> It confuses the devfreq governor and thermal governor (Intelligent Power
> >> Allocation (IPA)). Thermal governor works on stale stats data and makes
> >> stupid decisions, because there is no new stats (device not checked).
> >> Similar applies to devfreq simple_ondemand governor, where it 'tires' to
> >> work on a loooong period even 3sec and make prediction for the next
> >> frequency based on it (which is broken).
> >>
> >> How it should be done: constant reliable check is needed, then:
> >> - period is guaranteed and has fixed size, e.g 50ms or 100ms.
> >> - device status is quite recent so thermal devfreq cooling provides
> >> 'fresh' data into thermal governor
> >>
> >> This would prevent odd behavior and solve the broken cases.
> >>
> >>>
> >>> Considering it will wake up CPU more frequently, and user may choose
> >>> to change this by sysfs,
> >>> I'd like to not apply it.
> >>
> >> The deferred timer for GPU is wrong option, for UFS or eMMC makes more
> >> sense. It's also not recommended for NoC busses. I've discovered that
> >> some time ago and proposed to have option to switch into delayed timer.
> >> Trust me, it wasn't obvious to find out that this missing check has
> >> those impacts. So the other engineers or users might not know that some
> >> problems they faces (especially when the device load is changing) is due
> >> to this delayed vs deffered timer and they will change it in the sysfs.
> >>
> >> Regards,
> >> Lukasz
> >>
> >>>
> >>> Regards,
> >>> Qiang
> >>>
On 2021-02-03 02:01, Qiang Yu wrote:
> On Tue, Feb 2, 2021 at 10:02 PM Lukasz Luba <[email protected]> wrote:
>>
>>
>>
>> On 2/2/21 1:01 AM, Qiang Yu wrote:
>>> Hi Lukasz,
>>>
>>> Thanks for the explanation. So the deferred timer option makes a mistake that
>>> when GPU goes from idle to busy for only one poll periodic, in this
>>> case 50ms, right?
>>
>> Not exactly. Driver sets the polling interval to 50ms (in this case)
>> because it needs ~3-frame average load (in 60fps). I have discovered the
>> issue quite recently that on systems with 2 CPUs or more, the devfreq
>> core is not monitoring the devices even for seconds. Therefore, we might
>> end up with quite big amount of work that GPU is doing, but we don't
>> know about it. Devfreq core didn't check <- timer didn't fired. Then
>> suddenly that CPU, which had the deferred timer registered last time,
>> is waking up and timer triggers to check our device. We get the stats,
>> but they might be showing load from 1sec not 50ms. We feed them into
>> governor. Governor sees the new load, but was tested and configured for
>> 50ms, so it might try to rise the frequency to max. The GPU work might
>> be already lower and there is no need for such freq. Then the CPU goes
>> idle again, so no devfreq core check for next e.g. 1sec, but the
>> frequency stays at max OPP and we burn power.
>>
>> So, it's completely unreliable. We might stuck at min frequency and
>> suffer the frame drops, or sometimes stuck to max freq and burn more
>> power when there is no such need.
>>
>> Similar for thermal governor, which is confused by this old stats and
>> long period stats, longer than 50ms.
>>
>> Stats from last e.g. ~1sec tells you nothing about real recent GPU
>> workload.
> Oh, right, I missed this case.
>
>>
>>> But delayed timer will wakeup CPU every 50ms even when system is idle, will this
>>> cause more power consumption for the case like phone suspend?
>>
>> No, in case of phone suspend it won't increase the power consumption.
>> The device won't be woken up, it will stay in suspend.
> I mean the CPU is waked up frequently by timer when phone suspend,
> not the whole device (like the display).
>
> Seems it's better to have deferred timer when device is suspended for
> power saving,
> and delayed timer when device in working state. User knows this and
> can use sysfs
> to change it.
Doesn't devfreq_suspend_device() already cancel any timer work either
way in that case?
Robin.
> Set the delayed timer as default is reasonable, so patch is:
> Reviewed-by: Qiang Yu <[email protected]>
>
> Regards,
> Qiang
>
>>
>> Regards,
>> Lukasz
>>
>>
>>>
>>> Regards,
>>> Qiang
>>>
>>>
>>> On Mon, Feb 1, 2021 at 5:53 PM Lukasz Luba <[email protected]> wrote:
>>>>
>>>> Hi Qiang,
>>>>
>>>> On 1/30/21 1:51 PM, Qiang Yu wrote:
>>>>> Thanks for the patch. But I can't observe any difference on glmark2
>>>>> with or without this patch.
>>>>> Maybe you can provide other test which can benefit from it.
>>>>
>>>> This is a design problem and has impact on the whole system.
>>>> There is a few issues. When the device is not checked and there are
>>>> long delays between last check and current, the history is broken.
>>>> It confuses the devfreq governor and thermal governor (Intelligent Power
>>>> Allocation (IPA)). Thermal governor works on stale stats data and makes
>>>> stupid decisions, because there is no new stats (device not checked).
>>>> Similar applies to devfreq simple_ondemand governor, where it 'tires' to
>>>> work on a loooong period even 3sec and make prediction for the next
>>>> frequency based on it (which is broken).
>>>>
>>>> How it should be done: constant reliable check is needed, then:
>>>> - period is guaranteed and has fixed size, e.g 50ms or 100ms.
>>>> - device status is quite recent so thermal devfreq cooling provides
>>>> 'fresh' data into thermal governor
>>>>
>>>> This would prevent odd behavior and solve the broken cases.
>>>>
>>>>>
>>>>> Considering it will wake up CPU more frequently, and user may choose
>>>>> to change this by sysfs,
>>>>> I'd like to not apply it.
>>>>
>>>> The deferred timer for GPU is wrong option, for UFS or eMMC makes more
>>>> sense. It's also not recommended for NoC busses. I've discovered that
>>>> some time ago and proposed to have option to switch into delayed timer.
>>>> Trust me, it wasn't obvious to find out that this missing check has
>>>> those impacts. So the other engineers or users might not know that some
>>>> problems they faces (especially when the device load is changing) is due
>>>> to this delayed vs deffered timer and they will change it in the sysfs.
>>>>
>>>> Regards,
>>>> Lukasz
>>>>
>>>>>
>>>>> Regards,
>>>>> Qiang
>>>>>
> _______________________________________________
> dri-devel mailing list
> [email protected]
> https://lists.freedesktop.org/mailman/listinfo/dri-devel
>
On 2/4/21 1:39 PM, Robin Murphy wrote:
> On 2021-02-03 02:01, Qiang Yu wrote:
>> On Tue, Feb 2, 2021 at 10:02 PM Lukasz Luba <[email protected]> wrote:
>>>
>>>
>>>
>>> On 2/2/21 1:01 AM, Qiang Yu wrote:
>>>> Hi Lukasz,
>>>>
>>>> Thanks for the explanation. So the deferred timer option makes a
>>>> mistake that
>>>> when GPU goes from idle to busy for only one poll periodic, in this
>>>> case 50ms, right?
>>>
>>> Not exactly. Driver sets the polling interval to 50ms (in this case)
>>> because it needs ~3-frame average load (in 60fps). I have discovered the
>>> issue quite recently that on systems with 2 CPUs or more, the devfreq
>>> core is not monitoring the devices even for seconds. Therefore, we might
>>> end up with quite big amount of work that GPU is doing, but we don't
>>> know about it. Devfreq core didn't check <- timer didn't fired. Then
>>> suddenly that CPU, which had the deferred timer registered last time,
>>> is waking up and timer triggers to check our device. We get the stats,
>>> but they might be showing load from 1sec not 50ms. We feed them into
>>> governor. Governor sees the new load, but was tested and configured for
>>> 50ms, so it might try to rise the frequency to max. The GPU work might
>>> be already lower and there is no need for such freq. Then the CPU goes
>>> idle again, so no devfreq core check for next e.g. 1sec, but the
>>> frequency stays at max OPP and we burn power.
>>>
>>> So, it's completely unreliable. We might stuck at min frequency and
>>> suffer the frame drops, or sometimes stuck to max freq and burn more
>>> power when there is no such need.
>>>
>>> Similar for thermal governor, which is confused by this old stats and
>>> long period stats, longer than 50ms.
>>>
>>> Stats from last e.g. ~1sec tells you nothing about real recent GPU
>>> workload.
>> Oh, right, I missed this case.
>>
>>>
>>>> But delayed timer will wakeup CPU every 50ms even when system is
>>>> idle, will this
>>>> cause more power consumption for the case like phone suspend?
>>>
>>> No, in case of phone suspend it won't increase the power consumption.
>>> The device won't be woken up, it will stay in suspend.
>> I mean the CPU is waked up frequently by timer when phone suspend,
>> not the whole device (like the display).
>>
>> Seems it's better to have deferred timer when device is suspended for
>> power saving,
>> and delayed timer when device in working state. User knows this and
>> can use sysfs
>> to change it.
>
> Doesn't devfreq_suspend_device() already cancel any timer work either
> way in that case?
Correct, the governor should pause the monitoring mechanism (and timer).
Regards,
Lukasz
Applied to drm-misc-next.
Regards,
Qiang
On Thu, Feb 4, 2021 at 10:24 PM Lukasz Luba <[email protected]> wrote:
>
>
>
> On 2/4/21 1:39 PM, Robin Murphy wrote:
> > On 2021-02-03 02:01, Qiang Yu wrote:
> >> On Tue, Feb 2, 2021 at 10:02 PM Lukasz Luba <[email protected]> wrote:
> >>>
> >>>
> >>>
> >>> On 2/2/21 1:01 AM, Qiang Yu wrote:
> >>>> Hi Lukasz,
> >>>>
> >>>> Thanks for the explanation. So the deferred timer option makes a
> >>>> mistake that
> >>>> when GPU goes from idle to busy for only one poll periodic, in this
> >>>> case 50ms, right?
> >>>
> >>> Not exactly. Driver sets the polling interval to 50ms (in this case)
> >>> because it needs ~3-frame average load (in 60fps). I have discovered the
> >>> issue quite recently that on systems with 2 CPUs or more, the devfreq
> >>> core is not monitoring the devices even for seconds. Therefore, we might
> >>> end up with quite big amount of work that GPU is doing, but we don't
> >>> know about it. Devfreq core didn't check <- timer didn't fired. Then
> >>> suddenly that CPU, which had the deferred timer registered last time,
> >>> is waking up and timer triggers to check our device. We get the stats,
> >>> but they might be showing load from 1sec not 50ms. We feed them into
> >>> governor. Governor sees the new load, but was tested and configured for
> >>> 50ms, so it might try to rise the frequency to max. The GPU work might
> >>> be already lower and there is no need for such freq. Then the CPU goes
> >>> idle again, so no devfreq core check for next e.g. 1sec, but the
> >>> frequency stays at max OPP and we burn power.
> >>>
> >>> So, it's completely unreliable. We might stuck at min frequency and
> >>> suffer the frame drops, or sometimes stuck to max freq and burn more
> >>> power when there is no such need.
> >>>
> >>> Similar for thermal governor, which is confused by this old stats and
> >>> long period stats, longer than 50ms.
> >>>
> >>> Stats from last e.g. ~1sec tells you nothing about real recent GPU
> >>> workload.
> >> Oh, right, I missed this case.
> >>
> >>>
> >>>> But delayed timer will wakeup CPU every 50ms even when system is
> >>>> idle, will this
> >>>> cause more power consumption for the case like phone suspend?
> >>>
> >>> No, in case of phone suspend it won't increase the power consumption.
> >>> The device won't be woken up, it will stay in suspend.
> >> I mean the CPU is waked up frequently by timer when phone suspend,
> >> not the whole device (like the display).
> >>
> >> Seems it's better to have deferred timer when device is suspended for
> >> power saving,
> >> and delayed timer when device in working state. User knows this and
> >> can use sysfs
> >> to change it.
> >
> > Doesn't devfreq_suspend_device() already cancel any timer work either
> > way in that case?
>
> Correct, the governor should pause the monitoring mechanism (and timer).
>
> Regards,
> Lukasz