Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Sun, 7 May 2023 16:26:41 +0200
From:   Andrea Righi <andrea.righi@canonical.com>
To:     Steven Rostedt <rostedt@goodmis.org>
Cc:     Frederic Weisbecker <frederic@kernel.org>,
        Ingo Molnar <mingo@kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        "Rafael J. Wysocki" <rafael@kernel.org>,
        Jonathan Corbet <corbet@lwn.net>,
        Salvatore Bonaccorso <carnil@debian.org>,
        Pawan Gupta <pawan.kumar.gupta@linux.intel.com>,
        Mauro Carvalho Chehab <mchehab@kernel.org>,
        Daniel Lezcano <daniel.lezcano@linaro.org>,
        linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org,
        Anna-Maria Behnsen <anna-maria@linutronix.de>
Subject: Re: [PATCH] timers/nohz: introduce nohz_full_aggressive
Message-ID: <ZFe1IY+TdzSC0RLZ@righiandr-XPS-13-7390>
References: <20230507090700.18470-1-andrea.righi@canonical.com>
 <20230507100852.54a354ca@rorschach.local.home>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230507100852.54a354ca@rorschach.local.home>
Precedence: bulk

On Sun, May 07, 2023 at 10:08:52AM -0400, Steven Rostedt wrote:
> 
> [ Added Anna-Maria who is doing some timer work as well ]
> 
> On Sun,  7 May 2023 11:07:00 +0200
> Andrea Righi <andrea.righi@canonical.com> wrote:
> 
> > Overview:
> > 
> > nohz_full is a feature that allows to reduce the number of CPU tick
> > interrupts, thereby improving energy efficiency and reducing kernel
> > jitter.
> 
> Hmm, I never thought of NOHZ_FULL used for energy efficiency, as the
> CPU is still running user space code, and there's really nothing
> inherently more power consuming with the tick.

The idea here was to try to reduce the tick also on the timekeeping CPU
to have more idle time (because at least 1 CPU is periodically ticking
with nohz_full=all).

But my patch was mostly a toy patch and the real purpose was really to
get some advices/guidance on the tick/nohz topic.

> 
> > 
> > This works by stopping the tick interrupts on the CPUs that are either
> > idle or that have only one runnable task on them (there is no reason to
> > periodically interrupt the execution of a single running task if none
> > else is waiting to acquire the same CPU).
> > 
> > It is not possible to configure all the available CPUs to work in the
> > nohz_full mode, at least one non-adaptive-tick CPU must be periodically
> > interrupted to properly handle timekeeping tasks in the system (such as
> > the gettimeofday() syscall returning accurate values).
> 
> Do we really need nohz_full, instead, I think you want to look at what
> Anna-Maria is doing with moving the timer "manager" around to make sure
> that the tick stays on busy CPUs.
> 
> Again, nohz_full is not for power consumption savings, but instead to
> reduce kernel interruption in user space.

Will definitely look at Anna-Maria's work.

> 
> > 
> > However, under certain conditions, we may want to relax this constraint,
> > accepting potential time inaccuracies in the system, in order to provide
> > additional benefits in terms of power consumption, performance and/or
> > reduce kernel jitter even more.
> > 
> > For this reason introduce the new parameter nohz_full_aggressive.
> > 
> > This option allows to enforce nozh_full across all the CPUs (even the
> > timekeeping CPU) at the cost of having potential timer inaccuracies in
> > the system.
> > 
> > Test:
> > 
> >  - Hardware: Dell XPS 13 7390 w/ 8 cores
> > 
> >  - Kernel is using CONFIG_HZ=1000 (worst case scenario in terms of
> >    power consumption and kernel jitter) and nohz_full=all
> > 
> >  - Measure interrupts and power consumption when the system is idle and
> >    with 2, 4 and 8 cpu hogs
> > 
> > Result:
> > 
> > The following numbers have been collected using turbostat and dstat
> > measuring the average over a 5min run for each test.
> > 
> > irqs/sec             idle     1 CPU hog  2 CPU hogs  4 CPU hogs  8 CPU hogs
> >                      ------------------------------------------------------
> > nohz_full            1036.679  1047.522    1046.203    1048.590    1074.867
> > nohz_full_aggressive   98.685   106.296     127.587     146.586    1062.277
> > 
> > Power (Watt)         idle     1 CPU hog  2 CPU hogs  4 CPU hogs  8 CPU hogs
> >                      ------------------------------------------------------
> > nohz_full            0.502 W    3.436 W    3.755 W    6.187 W    6.019 W
> > nohz_full_aggressive 0.301 W    2.372 W    2.372 W    6.005 W    6.016 W
> > 
> > % power reduction     40.04%     30.97%     36.83%      2.94%      0.05%
> > 
> 
> Nice.
> 
> Now I doubt this is acceptable considering the side effects that the
> timer inaccuracy can cause. I think this breaks some basic assumptions
> in both the kernel and user space.

I've been running this nohz_full_aggressive patch for some days on my
laptop without any evident side effect, but I'm pretty sure it can break
something, considering that timing potentially can become totally
unreliable.

I was also wondering if we could try to implement a kind of dynamic HZ
scaling (like scaling HZ up/down dynamically at runtime or even at boot
time), but it seems quite complicated (and scary, especially looking at
the code in jiffies / timers, i.e. all the constants in
./kernel/time/timeconst.bc).

I remember there used to be a dynamic-hz patch a long long time ago by
Andrea Arcangeli, but I couldn't find any recent work on this topic.

> 
> Now, I think what is really happening here is that you are somewhat
> simulating the results that Anna-Maria has indirectly. That is, you
> just prevent an idle CPU from waking up to handle interrupts when not
> needed.
> 
> Anna-Maria,
> 
> Do you have some patches that Andrea could test with?
> 
> Thanks,
> 
> -- Steve

Thanks for looking at this (and I'm happy to help Anna-Maria with any
test).

-Andrea