Received-SPF: pass (google.com: domain of linux-kernel+bounces-189849-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Sun, 26 May 2024 21:27:58 +0200
From: Oleg Nesterov <oleg@redhat.com>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: Frederic Weisbecker <frederic@kernel.org>,
	Ingo Molnar <mingo@redhat.com>, Nicholas Piggin <npiggin@gmail.com>,
	Peter Zijlstra <peterz@infradead.org>, Phil Auld <pauld@redhat.com>,
	Chris von Recklinghausen <crecklin@redhat.com>,
	linux-kernel@vger.kernel.org
Subject: Re: sched/isolation: tick_take_do_timer_from_boot() calls
 smp_call_function_single() with irqs disabled
Message-ID: <20240526192758.GA21193@redhat.com>
References: <20240522151742.GA10400@redhat.com>
 <20240523132358.GA1965@redhat.com>
 <87h6eneeu7.ffs@tglx>
 <ZlCwKk65-eL0FrKX@pavilion.home>
 <20240524183700.GA17065@redhat.com>
 <87v832dfw1.ffs@tglx>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87v832dfw1.ffs@tglx>
User-Agent: Mutt/1.5.24 (2015-08-30)

The more I grep the more I confused.

On 05/25, Thomas Gleixner wrote:
>
> Right. It does not happen because the kernel starts with jiffies as
> clocksource except on S390. The jiffies clocksource is not qualified to
> switch over to NOHZ mode for obvious reasons.

Not obvious for those who never looked at this code ;)

OK, clocksource_jiffies doesn't have CLOCK_SOURCE_VALID_FOR_HRES,

> fs_initcall(clocksource_done_booting)

So clocksource_done_booting() -> clocksource_select() should find another
CLOCK_SOURCE_VALID_FOR_HRES cs, then timekeeping_notify(best)

	- sets tk_core.timekeeper.tkr_mono.clock = best

	- calls tick_clock_notify() which sets .check_clocks on every cpu.

This makes tick_check_oneshot_change(false) return true. (I am ignoring the
highres=n case, tick_nohz_switch_to_nohz() doesn't really differ).

Now, on every CPU the next ->event_handler == tick_handle_periodic path
call hrtimer_switch_to_hres() which

	- sets ->event_handler == hrtimer_interrupt (tick_init_highres)
	  so tick_periodic/do_timer will be never called again

	- calls tick_setup_sched_timer() -> tick_nohz_activate() which
	  sets TS_FLAG_NOHZ

> [    1.918548] clocksource_done_booting: Switched to NOHZ // debug printk
>
> This is the point where tick_nohz_activate() is called first time and
> that does:
>
>   tick_sched_flag_set(ts, TS_FLAG_NOHZ);

See above, but I got lost, most probably I misunderstood these nontrivial
code paths.

> So up to this point the tick is never stopped neither on housekeeping
> nor on NOHZ FULL CPUs:
>
> tick_nohz_full_update_tick()
>   if (!tick_sched_flag_test(ts, TS_FLAG_NOHZ))
>     return;

OK... But tick_nohz_idle_update_tick() doesn't check TS_FLAG_NOHZ and
the tick_nohz_full_cpu() check can't help at boot time.

And I still don't understand why we can rely on can_stop_idle_tick() even
in tick_nohz_idle_stop_tick().

I'll try to read this code again tomorrow, but it will never fit my poor
little brain ;)

Oleg.