We are trying to implement a solution for thermal mitigation by using
idle injection on CPUs. However we face some limitations with the
current idle-inject framework. As per our need, we want to start
injecting idle cycles on a cpu for indefinite time (until the
temperature/power of the CPU falls below a threshold). This will allow
to keep the hot CPUs in the sleep state until we see improvement in
temperature/power. If we set idle duration to a large value or have an
idle-injection ratio of 100%, then the idle-inject RT thread suffers
from RT throttling. This results in the CPU exiting from the sleep state
and consume some power.
To solve this limitation, we propose a solution to disable RT-throttling
whenever idle-inject threads run. We achieve this by not accounting the
runtime for the idle-inject threads.
Atul Pant (2):
sched/rt: Disable runtime accounting for idle threads with SCHED_FIFO
policy
sched/idle: Add a description for play_idle_precise
kernel/sched/idle.c | 5 +++++
kernel/sched/rt.c | 4 +++-
2 files changed, 8 insertions(+), 1 deletion(-)
--
2.25.1
Since runtime accounting of any RT idle thread is prevented, add a
description about this function that it runs idle loops on a cpu
without these threads being preempted due to RT throttling.
Signed-off-by: Atul Pant <[email protected]>
---
kernel/sched/idle.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
index 5007b25c5bc6..7aaf3679f50a 100644
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -334,6 +334,11 @@ static enum hrtimer_restart idle_inject_timer_fn(struct hrtimer *timer)
return HRTIMER_NORESTART;
}
+/*
+ * This function runs idle loops on a cpu, whose policy is set as SCHED_FIFO.
+ * Despite being a rt thread, runtime for it is not accounted, thus preventing
+ * its preemption due to rt-throttling
+ */
void play_idle_precise(u64 duration_ns, u64 latency_ns)
{
struct idle_timer it;
--
2.25.1
On Wed, Apr 10, 2024 at 10:54:41AM +0200, Peter Zijlstra wrote:
> On Wed, Apr 10, 2024 at 10:24:15AM +0530, Atul Pant wrote:
> > We are trying to implement a solution for thermal mitigation by using
> > idle injection on CPUs. However we face some limitations with the
> > current idle-inject framework. As per our need, we want to start
> > injecting idle cycles on a cpu for indefinite time (until the
> > temperature/power of the CPU falls below a threshold). This will allow
> > to keep the hot CPUs in the sleep state until we see improvement in
> > temperature/power. If we set idle duration to a large value or have an
> > idle-injection ratio of 100%, then the idle-inject RT thread suffers
> > from RT throttling. This results in the CPU exiting from the sleep state
> > and consume some power.
> >
> > To solve this limitation, we propose a solution to disable RT-throttling
> > whenever idle-inject threads run. We achieve this by not accounting the
> > runtime for the idle-inject threads.
>
> Running RT tasks for indefinite amounts of time will wreck the system.
> Things like workqueues and other per-cpu threads expect service or
> things will pile up and run to ground.
>
> Idle injection, just like every other RT user must not be able to starve
> the system of service.
>
> If your system design requires this (I would argue it is broken), look
> at other means, like CPU-hotplug (which I also really detest) -- which
> takes down the CPU in a controlled manner and avoids the resource
> issues.
Hi Peter,
We are trying to add support for true 100% idle-injection ratio from
idle-injection framework. It might happen that we want to run idle cycles for
slightly more time than permitted by RT-bandwidth control. We understand the
concern about it hogging the cpu. Will it be better if we make it a choice for
the user who uses idle-inject framework, whether to have true 100%
idle-injection support or not?
Thanks
Atul
On Wed, Apr 10, 2024 at 04:59:33PM +0530, Atul Kumar Pant wrote:
> Hi Peter,
> We are trying to add support for true 100% idle-injection ratio from
> idle-injection framework.
Yeah, I got that. I'm saying that is broken. Both from a requirement POV
and an implementation POV.
If your hardware needs 100% idle injection that means CPU availability
is unreliable and everything that relies on CPU-masks will be broken.
Furthermore, since idle-injection is build on top of FIFO, anything with
a higher priority (DL, stop) will simply preempt it anyway.
And, as already mentioned, hogging the system with FIFO will break
things.
So I would *very* strongly urge you to push back on whoever thought this
is 'needed'. This is *bad* hardware.
> It might happen that we want to run idle cycles for
> slightly more time than permitted by RT-bandwidth control.
The thing is configurable for a reason.
> We understand the
> concern about it hogging the cpu. Will it be better if we make it a choice for
> the user who uses idle-inject framework, whether to have true 100%
> idle-injection support or not?
None of the above mentioned issues will magically go away. Run a 100%
FIFO task for an indeterminate amount of time and you get to keep the
pieces.
Also, we'll be replacing the throttling code with DL servers 'soonish'
at which point all this will stop working anyway, since DL will preempt
anything FIFO, including your idle injection crud.
On 4/10/24 13:46, Peter Zijlstra wrote:
> Also, we'll be replacing the throttling code with DL servers 'soonish'
> at which point all this will stop working anyway, since DL will preempt
> anything FIFO, including your idle injection crud.
+1
also, given that the code spins with preempt disabled, with dl server it could
even become a non-rt thread...
FIFO RUNNING
DL_SERVER activates
their loop
disables preemption()
run()
enable preemption()
DL_SERVE throttled
FIFO BACK
So, there will be no need for this busy loop to be RT.
Anyways, it breaks RT and DL if it keeps running for too long... It can
also cause complaints like RCU stalls and loong wait on locks, e.g., on
kworkers...
-- Daniel
On Wed, 10 Apr 2024 10:24:15 +0530
Atul Pant <[email protected]> wrote:
> We are trying to implement a solution for thermal mitigation by using
> idle injection on CPUs. However we face some limitations with the
> current idle-inject framework. As per our need, we want to start
> injecting idle cycles on a cpu for indefinite time (until the
> temperature/power of the CPU falls below a threshold). This will allow
> to keep the hot CPUs in the sleep state until we see improvement in
> temperature/power. If we set idle duration to a large value or have an
> idle-injection ratio of 100%, then the idle-inject RT thread suffers
> from RT throttling. This results in the CPU exiting from the sleep state
> and consume some power.
>
> To solve this limitation, we propose a solution to disable RT-throttling
> whenever idle-inject threads run. We achieve this by not accounting the
> runtime for the idle-inject threads.
I'm going to assume that when dl-server is finally accepted, this will no
longer be an issue for you?
https://lore.kernel.org/all/[email protected]/
-- Steve
On Wed, 10 Apr 2024 at 14:24, Daniel Bristot de Oliveira
<[email protected]> wrote:
>
> On 4/10/24 13:46, Peter Zijlstra wrote:
> > Also, we'll be replacing the throttling code with DL servers 'soonish'
> > at which point all this will stop working anyway, since DL will preempt
> > anything FIFO, including your idle injection crud.
The DL server could make things better for idle injection as it will
let the idle RT threads running on the CPU as long as there is no cfs
thread enqueued whereas the current rt throttling force throttling RT
even if there is no enqueued cfs task. So as long as you have been
able to move things on other cpus, your idle rt thread should keep
running.
>
> +1
>
> also, given that the code spins with preempt disabled, with dl server it could
> even become a non-rt thread...
>
> FIFO RUNNING
> DL_SERVER activates
> their loop
> disables preemption()
> run()
> enable preemption()
> DL_SERVE throttled
> FIFO BACK
>
> So, there will be no need for this busy loop to be RT.
>
> Anyways, it breaks RT and DL if it keeps running for too long... It can
> also cause complaints like RCU stalls and loong wait on locks, e.g., on
> kworkers...
>
> -- Daniel
>
>
>
>
>
On Wed, Apr 10, 2024 at 10:24:15AM +0530, Atul Pant wrote:
> We are trying to implement a solution for thermal mitigation by using
> idle injection on CPUs. However we face some limitations with the
> current idle-inject framework. As per our need, we want to start
> injecting idle cycles on a cpu for indefinite time (until the
> temperature/power of the CPU falls below a threshold). This will allow
> to keep the hot CPUs in the sleep state until we see improvement in
> temperature/power. If we set idle duration to a large value or have an
> idle-injection ratio of 100%, then the idle-inject RT thread suffers
> from RT throttling. This results in the CPU exiting from the sleep state
> and consume some power.
>
> To solve this limitation, we propose a solution to disable RT-throttling
> whenever idle-inject threads run. We achieve this by not accounting the
> runtime for the idle-inject threads.
Running RT tasks for indefinite amounts of time will wreck the system.
Things like workqueues and other per-cpu threads expect service or
things will pile up and run to ground.
Idle injection, just like every other RT user must not be able to starve
the system of service.
If your system design requires this (I would argue it is broken), look
at other means, like CPU-hotplug (which I also really detest) -- which
takes down the CPU in a controlled manner and avoids the resource
issues.