Received-SPF: pass (google.com: domain of linux-kernel+bounces-186422-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Wed, 22 May 2024 17:17:42 +0200
From: Oleg Nesterov <oleg@redhat.com>
To: Frederic Weisbecker <frederic@kernel.org>,
	Ingo Molnar <mingo@redhat.com>, Nicholas Piggin <npiggin@gmail.com>,
	Peter Zijlstra <peterz@infradead.org>, Phil Auld <pauld@redhat.com>,
	Thomas Gleixner <tglx@linutronix.de>
Cc: Chris von Recklinghausen <crecklin@redhat.com>,
	linux-kernel@vger.kernel.org
Subject: sched/isolation: tick_take_do_timer_from_boot() calls
 smp_call_function_single() with irqs disabled
Message-ID: <20240522151742.GA10400@redhat.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.24 (2015-08-30)

After the recent comment 5097cbcb38e6 ("sched/isolation: Prevent boot crash
when the boot CPU is nohz_full") the kernel no longer crashes, but there is
another problem.

In this case tick_setup_device() does tick_take_do_timer_from_boot() to
update tick_do_timer_cpu and this triggers WARN_ON_ONCE(irqs_disabled())
in smp_call_function_single().

I don't understand this code even remotely, I failed to find the fix.

Perhaps we can use smp_call_function_single_async() as a workaround ?

But I don't even understand why exactly we need smp_call_function()...
Lets suppose we change

	--- a/kernel/time/tick-common.c
	+++ b/kernel/time/tick-common.c
	@@ -231,9 +231,9 @@ static void tick_setup_device(struct tick_device *td,
	 
			} else if (tick_do_timer_boot_cpu != -1 &&
							!tick_nohz_full_cpu(cpu)) {
	-			tick_take_do_timer_from_boot();
	+//			tick_take_do_timer_from_boot();
				tick_do_timer_boot_cpu = -1;
	-			WARN_ON(READ_ONCE(tick_do_timer_cpu) != cpu);
	+			WRITE_ONCE(tick_do_timer_cpu), cpu);
	 #endif
			}

Can anyone explain what exactly can go wrong?

Race with tick_nohz_stop_tick() on boot CPU which can set
tick_do_timer_cpu = TICK_DO_TIMER_NONE? Is it really bad?

Something else?

Please help, I don't think I can find a sane solution :/

Oleg.