Received-SPF: pass (google.com: domain of linux-kernel+bounces-191601-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Precedence: bulk
Mime-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8
Date: Tue, 28 May 2024 11:02:03 +1000
Message-Id: <D1KVETKKSRHL.18ZTVKAN8JS3Y@gmail.com>
From: "Nicholas Piggin" <npiggin@gmail.com>
To: "Oleg Nesterov" <oleg@redhat.com>
Cc: "Thomas Gleixner" <tglx@linutronix.de>, "Frederic Weisbecker"
 <frederic@kernel.org>, "Ingo Molnar" <mingo@redhat.com>, "Peter Zijlstra"
 <peterz@infradead.org>, "Phil Auld" <pauld@redhat.com>, "Chris von
 Recklinghausen" <crecklin@redhat.com>, <linux-kernel@vger.kernel.org>
Subject: Re: sched/isolation: tick_take_do_timer_from_boot() calls
 smp_call_function_single() with irqs disabled
References: <20240522151742.GA10400@redhat.com>
 <20240523132358.GA1965@redhat.com> <87h6eneeu7.ffs@tglx>
 <ZlCwKk65-eL0FrKX@pavilion.home> <20240524183700.GA17065@redhat.com>
 <87v832dfw1.ffs@tglx> <20240526192758.GA21193@redhat.com>
 <D1KDJILBPEKS.27MVR6A44NMX8@gmail.com> <20240527155739.GB5733@redhat.com>
In-Reply-To: <20240527155739.GB5733@redhat.com>

On Tue May 28, 2024 at 1:57 AM AEST, Oleg Nesterov wrote:
> On 05/27, Nicholas Piggin wrote:
> >
> > On Mon May 27, 2024 at 5:27 AM AEST, Oleg Nesterov wrote:
> > > The more I grep the more I confused.
> > >
> > > On 05/25, Thomas Gleixner wrote:
> > > >
> > > > Right. It does not happen because the kernel starts with jiffies as
> > > > clocksource except on S390. The jiffies clocksource is not qualifie=
d to
> > > > switch over to NOHZ mode for obvious reasons.
> > >
> > > Not obvious for those who never looked at this code ;)
> > >
> > > OK, clocksource_jiffies doesn't have CLOCK_SOURCE_VALID_FOR_HRES,
> >
> > jiffies clocksource requires a timer to run it I suppose.
>
> I meant, this is enough to ensure that clocksource_done_booting() paths s=
hould
> find another cs, best !=3D curr_clocksource, and call timekeeping_notify(=
).
>
> Nevermind, quite possibly I misread this code.
>
> > > And I still don't understand why we can rely on can_stop_idle_tick() =
even
> > > in tick_nohz_idle_stop_tick().
> >
> > AFAIKS timer_expires_base would be 0 unless tick_nohz_next_event()
> > were called, but that is only called from places that checked
> > can_stop_idle_tick() or is already a tick_nohz_full() CPU.
>
> OK, thanks.
>
> So, Frederic, Nicholas, any objections to the trivial change below?

Since Thomas says it's alright, then no. I guess I added it because I
was not certain about taking the tick_do_timer_cpu while the boot CPU
could be running a timer interrupt.

I would take some of his comment to explain the race is harmless and
put it in that if block.

Out of curiosity, you are getting this going on x86? Or another arch?
Any particular use-case?

Thanks,
Nick

>
> We can cleanup the tick_do_timer_boot_cpu logic on top of it.
>
> Oleg.
>
> --- a/kernel/time/tick-common.c
> +++ b/kernel/time/tick-common.c
> @@ -178,26 +178,6 @@ void tick_setup_periodic(struct clock_event_device *=
dev, int broadcast)
>  	}
>  }
> =20
> -#ifdef CONFIG_NO_HZ_FULL
> -static void giveup_do_timer(void *info)
> -{
> -	int cpu =3D *(unsigned int *)info;
> -
> -	WARN_ON(tick_do_timer_cpu !=3D smp_processor_id());
> -
> -	tick_do_timer_cpu =3D cpu;
> -}
> -
> -static void tick_take_do_timer_from_boot(void)
> -{
> -	int cpu =3D smp_processor_id();
> -	int from =3D tick_do_timer_boot_cpu;
> -
> -	if (from >=3D 0 && from !=3D cpu)
> -		smp_call_function_single(from, giveup_do_timer, &cpu, 1);
> -}
> -#endif
> -
>  /*
>   * Setup the tick device
>   */
> @@ -231,9 +211,8 @@ static void tick_setup_device(struct tick_device *td,
> =20
>  		} else if (tick_do_timer_boot_cpu !=3D -1 &&
>  						!tick_nohz_full_cpu(cpu)) {
> -			tick_take_do_timer_from_boot();
>  			tick_do_timer_boot_cpu =3D -1;
> -			WARN_ON(READ_ONCE(tick_do_timer_cpu) !=3D cpu);
> +			WRITE_ONCE(tick_do_timer_cpu, cpu);
>  #endif
>  		}
> =20