Received-SPF: pass (google.com: domain of linux-kernel+bounces-48608-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Thu, 1 Feb 2024 18:43:03 +0100
From: Frederic Weisbecker <frederic@kernel.org>
To: Anna-Maria Behnsen <anna-maria@linutronix.de>
Cc: linux-kernel@vger.kernel.org, Peter Zijlstra <peterz@infradead.org>,
	John Stultz <jstultz@google.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Eric Dumazet <edumazet@google.com>,
	"Rafael J . Wysocki" <rafael.j.wysocki@intel.com>,
	Arjan van de Ven <arjan@infradead.org>,
	"Paul E . McKenney" <paulmck@kernel.org>,
	Rik van Riel <riel@surriel.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Sebastian Siewior <bigeasy@linutronix.de>,
	Giovanni Gherdovich <ggherdovich@suse.cz>,
	Lukasz Luba <lukasz.luba@arm.com>,
	"Gautham R . Shenoy" <gautham.shenoy@amd.com>,
	Srinivas Pandruvada <srinivas.pandruvada@intel.com>,
	K Prateek Nayak <kprateek.nayak@amd.com>
Subject: Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model
Message-ID: <ZbvYJysR7gnaQiNg@localhost.localdomain>
References: <20240115143743.27827-1-anna-maria@linutronix.de>
 <20240115143743.27827-19-anna-maria@linutronix.de>
 <ZbuzMeKlCgebhJZ-@localhost.localdomain>
 <87bk90i2ja.fsf@somnus>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <87bk90i2ja.fsf@somnus>

Le Thu, Feb 01, 2024 at 05:15:37PM +0100, Anna-Maria Behnsen a ?crit :
> Frederic Weisbecker <frederic@kernel.org> writes:
> 
> > Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a ?crit :
> >> +static void tmigr_connect_child_parent(struct tmigr_group *child,
> >> +				       struct tmigr_group *parent)
> >> +{
> >> +	union tmigr_state childstate;
> >> +
> >> +	raw_spin_lock_irq(&child->lock);
> >> +	raw_spin_lock_nested(&parent->lock, SINGLE_DEPTH_NESTING);
> >> +
> >> +	child->parent = parent;
> >> +	child->childmask = BIT(parent->num_children++);
> >> +
> >> +	raw_spin_unlock(&parent->lock);
> >> +	raw_spin_unlock_irq(&child->lock);
> >> +
> >> +	/*
> >> +	 * To prevent inconsistent states, active children need to be active in
> >> +	 * the new parent as well. Inactive children are already marked inactive
> >> +	 * in the parent group.
> >> +	 */
> >> +	childstate.state = atomic_read(&child->migr_state);
> >> +	if (childstate.migrator != TMIGR_NONE) {
> >
> > Is it possible here to connect a running online child (not one that we just
> > created) to a new parent?
> 
> connect_child_parent() is only executed for the just created ones. So,
> yes in theory this would be possible, but it doesn't happen as
> tmigr_setup_groups() takes care to make it right (hopefully :)). When a
> LVL0 group has some space left, only the connection between tmc and the
> LVL0 group is done in tmigr_setup_groups(). If there is no space left in
> LVL0 group, then a new group is created and depending on the levels
> which has to be created only executed for the new ones.
> 
> > If not, is it possible that a newly created child is
> > not TMIGR_NONE?
> 
> Yes. See tmigr_cpu_online(). When new groups have to be created starting
> from LVL0, then they are not active - so TMIGR_NONE is set. Activating
> the new online CPU is done afterwards.
> 
> But if it is required to add also a new level at the top, then it is
> mandatory to propagate the active state of the already existing child to
> the new parent. The connect_child_parent() is then also executed for the
> formerly top level group (child) to the newly created group (parent).

Ah and this is why we have the "if (childstate.migrator != TMIGR_NONE)"
branch, right?

> > Heh, I was about to say that it's impossible that timer_base_is_idle()
> > at this stage but actually if we run in nohz_full...
> >
> > It happens so that nohz_full is deactivated until rcutree_online_cpu()
> > which calls tick_dep_clear() but it's a pure coincidence that might
> > disappear one day. So yes, let's keep it that way.
> 
> I instrumented the code (with NOHZ FULL and NOHZ_IDLE) to make sure the
> timer migration hierarchy state 'idle' is in sync with the timer base
> 'idle'. And this was one part where it was possible that it runs out of
> sync as I remember correctly. But if I understood you correctly, this
> shouldn't happen at the moment?

Well, it's not supposed to :-)

Thanks.

> 
> Thanks,
> 
> 	Anna-Maria
>