Received-SPF: pass (google.com: domain of linux-kernel+bounces-190292-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Precedence: bulk
Mime-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8
Date: Mon, 27 May 2024 19:10:07 +1000
Message-Id: <D1KB5Z1455DI.3HOVYR566ZGXN@gmail.com>
To: "Thomas Gleixner" <tglx@linutronix.de>, "Oleg Nesterov"
 <oleg@redhat.com>, "Frederic Weisbecker" <frederic@kernel.org>
Cc: "Ingo Molnar" <mingo@redhat.com>, "Peter Zijlstra"
 <peterz@infradead.org>, "Phil Auld" <pauld@redhat.com>, "Chris von
 Recklinghausen" <crecklin@redhat.com>, <linux-kernel@vger.kernel.org>
Subject: Re: sched/isolation: tick_take_do_timer_from_boot() calls
 smp_call_function_single() with irqs disabled
From: "Nicholas Piggin" <npiggin@gmail.com>
References: <20240522151742.GA10400@redhat.com>
 <20240523132358.GA1965@redhat.com> <87h6eneeu7.ffs@tglx>
 <ZlCwKk65-eL0FrKX@pavilion.home> <20240524183700.GA17065@redhat.com>
 <87v832dfw1.ffs@tglx>
In-Reply-To: <87v832dfw1.ffs@tglx>

On Sat May 25, 2024 at 8:06 AM AEST, Thomas Gleixner wrote:
> On Fri, May 24 2024 at 20:37, Oleg Nesterov wrote:
>
> > I've already had a few beers today, I know I'll regret about this
> > email tomorrow, but I can't resist ;)
>
> You won't regret it. :)
>
> > On 05/24, Frederic Weisbecker wrote:
> > But again, again. tick_sched_do_timer() says
> >
> > 	* If nohz_full is enabled, this should not happen because the
> > 	* 'tick_do_timer_cpu' CPU never relinquishes.
> >
> > so I guess it is not supposed to happen?
>
> Right. It does not happen because the kernel starts with jiffies as
> clocksource except on S390. The jiffies clocksource is not qualified to
> switch over to NOHZ mode for obvious reasons. But even on S390 which has
> a truly usable and useful clocksource the tick stays periodic to begin
> with. Why?
>
> The NOHZ ready notification happens late in the boot process via:
> fs_initcall(clocksource_done_booting)
>
> So by the time that happens, the secondary CPUs are up and have taken
> over the do timer duty.
>
> [    0.600381] smp: Bringing up secondary CPUs ...
>
> ....
>
> [    1.917842] clocksource: Switched to clocksource kvm-clock
> [    1.918548] clocksource_done_booting: Switched to NOHZ // debug printk
>
> This is the point where tick_nohz_activate() is called first time and
> that does:
>
>   tick_sched_flag_set(ts, TS_FLAG_NOHZ);
>
> So up to this point the tick is never stopped neither on housekeeping
> nor on NOHZ FULL CPUs:
>
> tick_nohz_full_update_tick()
>   if (!tick_sched_flag_test(ts, TS_FLAG_NOHZ))
>     return;
>
> > And. My main question was: how can smp_call_function_single() help???
>
> It's useless.
>
> > Why do we actually need it?
>
> We do not.
>
> As explained above there is also nothing extra to fix contrary to
> Frederics fears.
>
> Even in the case that a command line limitation restricts the number of
> CPUs such that there is no housekeeping CPU onlined during
> smp_init(). That is checked in the isolation init code which clears
> nohz_full_running in that case. Nothing to see there either.
>
> So all this needs is the simple:
>
> diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c
> index d88b13076b79..dab17d756fd8 100644
> --- a/kernel/time/tick-common.c
> +++ b/kernel/time/tick-common.c
> @@ -229,11 +209,9 @@ static void tick_setup_device(struct tick_device *td=
,
>  			if (tick_nohz_full_cpu(cpu))
>  				tick_do_timer_boot_cpu =3D cpu;
> =20
> -		} else if (tick_do_timer_boot_cpu !=3D -1 &&
> -						!tick_nohz_full_cpu(cpu)) {
> -			tick_take_do_timer_from_boot();
> +		} else if (tick_do_timer_boot_cpu !=3D -1 && !tick_nohz_full_cpu(cpu))=
 {
> +			WRITE_ONCE(tick_do_timer_cpu, cpu);
>  			tick_do_timer_boot_cpu =3D -1;
> -			WARN_ON(READ_ONCE(tick_do_timer_cpu) !=3D cpu);
>  #endif
>  		}
>
> along with the removal of the SMP function call voodoo programming gunk,
> a lengthy changelog and a bunch of useful comments.

I might not have tested that path on powerpc since it should not
switch clockevent driver (or clocksource either I think) at least
on 64-bit.  Explains the smp_call_function warning if you are
testing on x86 :/

Thanks,
Nick