Received-SPF: pass (google.com: domain of linux-kernel+bounces-123584-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
From: Valentin Schneider <vschneid@redhat.com>
To: Frederic Weisbecker <frederic@kernel.org>
Cc: "Paul E. McKenney" <paulmck@kernel.org>, Thomas Gleixner
 <tglx@linutronix.de>, LKML <linux-kernel@vger.kernel.org>, Ingo Molnar
 <mingo@kernel.org>, Anna-Maria Behnsen <anna-maria@linutronix.de>, Alex
 Shi <alexs@kernel.org>, Peter Zijlstra <peterz@infradead.org>, Vincent
 Guittot <vincent.guittot@linaro.org>, Barry Song
 <song.bao.hua@hisilicon.com>
Subject: Re: for_each_domain()/sched_domain_span() has offline CPUs (was Re:
 [PATCH 2/2] timers: Fix removed self-IPI on global timer's enqueue in
 nohz_full)
In-Reply-To: <ZgWhteHzLb8Jutp3@localhost.localdomain>
References: <ZfsLtMijRrNZfqh6@localhost.localdomain>
 <6a95b6ac-6681-4492-b155-e30c19bb3341@paulmck-laptop>
 <ZfwdEROGFFmIbkCM@lothringen>
 <bf8689c2-0749-47cb-9535-53cf66e34f5e@paulmck-laptop>
 <Zf1sSv/6mQHJuJ8G@lothringen> <Zf2GDjekyemQKn7I@lothringen>
 <xhsmhwmppaqls.mognet@vschneid-thinkpadt14sgen2i.remote.csb>
 <ZgQULtxy0UnoXfGi@localhost.localdomain>
 <xhsmhttkrbvfb.mognet@vschneid-thinkpadt14sgen2i.remote.csb>
 <xhsmhr0fubgaf.mognet@vschneid-thinkpadt14sgen2i.remote.csb>
 <ZgWhteHzLb8Jutp3@localhost.localdomain>
Date: Thu, 28 Mar 2024 21:31:21 +0100
Message-ID: <xhsmhmsqiayjq.mognet@vschneid-thinkpadt14sgen2i.remote.csb>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 28/03/24 17:58, Frederic Weisbecker wrote:
> Le Thu, Mar 28, 2024 at 03:08:08PM +0100, Valentin Schneider a =C3=A9crit=
 :
>> On 27/03/24 15:28, Valentin Schneider wrote:
>> > On 27/03/24 13:42, Frederic Weisbecker wrote:
>> >> Le Tue, Mar 26, 2024 at 05:46:07PM +0100, Valentin Schneider a =C3=A9=
crit :
>> >>> > Then with that patch I ran TREE07, just some short iterations:
>> >>> >
>> >>> > tools/testing/selftests/rcutorture/bin/kvm.sh --configs "10*TREE07=
" --allcpus --bootargs "rcutorture.onoff_interval=3D200" --duration 2
>> >>> >
>> >>> > And the warning triggers very quickly. At least since v6.3 but may=
be since
>> >>> > earlier. Is this expected behaviour or am I right to assume that
>> >>> > for_each_domain()/sched_domain_span() shouldn't return an offline =
CPU?
>> >>> >
>> >>>=20
>> >>> I would very much assume an offline CPU shouldn't show up in a
>> >>> sched_domain_span().
>> >>>=20
>> >>> Now, on top of the above, there's one more thing worth noting:
>> >>>   cpu_up_down_serialize_trainwrecks()
>> >>>=20
>> >>> This just flushes the cpuset work, so after that the sched_domain to=
pology
>> >>> should be sane. However I see it's invoked at the tail end of _cpu_d=
own(),
>> >>> IOW /after/ takedown_cpu() has run, which sounds too late. The comme=
nts
>> >>> around this vs. lock ordering aren't very reassuring however, so I n=
eed to
>> >>> look into this more.
>> >>
>> >> Ouch...
>> >>
>> >>>=20
>> >>> Maybe as a "quick" test to see if this is the right culprit, you cou=
ld try
>> >>> that with CONFIG_CPUSET=3Dn? Because in that case the sched_domain u=
pdate is
>> >>> ran within sched_cpu_deactivate().
>> >>
>> >> I just tried and I fear that doesn't help. It still triggers even wit=
hout
>> >> cpusets :-s
>> >>
>> >
>> > What, you mean I can't always blame cgroups? What has the world come t=
o?
>> >
>> > That's interesting, it means the deferred work item isn't the (only)
>> > issue. I'll grab your test patch and try to reproduce on TREE07.
>> >
>>=20
>> Unfortunately I haven't been able to trigger your warning with ~20 runs =
of
>> TREE07 & CONFIG_CPUSETS=3Dn, however it does trigger reliably with
>> CONFIG_CPUSETS=3Dy, so I'm back to thinking the cpuset work is a likely
>> culprit...
>
> Funny, I just checked again and I can still reliably reproduce with:
>
> ./tools/testing/selftests/rcutorture/bin/kvm.sh --kconfig "CONFIG_CPUSETS=
=3Dn CONFIG_PROC_PID_CPUSET=3Dn" --configs "10*TREE07" --allcpus --bootargs=
 "rcutorture.onoff_interval=3D200" --duration 2
>
> I'm thinking there might be several culprits... ;-)

Hmm, frustrating that I can't seem to reproduce this...

Could you run this with CONFIG_SCHED_DEBUG=3Dy and sched_verbose on the
cmdline? And maybe tweak the warning to show which CPU we are scanning the
sched_domain of and which one we found to be offline in the span.