Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Valentin Schneider <vschneid@redhat.com>
To:     paulmck@kernel.org
Cc:     linux-kernel@vger.kernel.org, mingo@redhat.com,
        peterz@infradead.org, juri.lelli@redhat.com,
        vincent.guittot@linaro.org, dietmar.eggemann@arm.com,
        rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de,
        bristot@redhat.com
Subject: Re: "Dying CPU not properly vacated" splat
In-Reply-To: <20220425173320.GX4285@paulmck-ThinkPad-P17-Gen-1>
References: <20220421193821.GA173010@paulmck-ThinkPad-P17-Gen-1>
 <xhsmh4k2h9m26.mognet@vschneid.remote.csb>
 <20220425173320.GX4285@paulmck-ThinkPad-P17-Gen-1>
Date:   Mon, 25 Apr 2022 22:59:44 +0100
Message-ID: <xhsmh1qxkakof.mognet@vschneid.remote.csb>
MIME-Version: 1.0
Content-Type: text/plain
Precedence: bulk

On 25/04/22 10:33, Paul E. McKenney wrote:
> On Mon, Apr 25, 2022 at 05:15:13PM +0100, Valentin Schneider wrote:
>>
>> Hi Paul,
>>
>> On 21/04/22 12:38, Paul E. McKenney wrote:
>> > Hello!
>> >
>> > The rcutorture TREE03 scenario got the following splat, which appears
>> > to be a one-off, or if not, having an MTBF in the thousands of hours,
>> > even assuming that it is specific to TREE03.  (If it is not specific to
>> > TREE03, we are talking tens of thousands of hours of rcutorture runtime.)
>> >
>> > So just in case this rings any bells or there are some diagnostics I
>> > should add in case this ever happens again.  ;-)
>>
>> There should be a dump of the enqueued tasks right after the snippet you've
>> sent, any chance you could share that if it's there? That should tell us
>> which tasks are potentially misbehaving.
>
> And now that I know to look for them, there they are!  Thank you!!!
>
> CPU7 enqueued tasks (2 total):
>  pid: 52, name: migration/7
>  pid: 135, name: rcu_torture_rea
> smpboot: CPU 7 is now offline
>
> So what did rcu_torture_reader() do wrong here?  ;-)
>

So on teardown, CPUHP_AP_SCHED_WAIT_EMPTY->sched_cpu_wait_empty() waits for
the rq to be empty. Tasks must *not* be enqueued onto that CPU after that
step has been run - if there are per-CPU tasks bound to that CPU, they must
be unbound in their respective hotplug callback.

For instance for workqueue.c, we have workqueue_offline_cpu() as a hotplug
callback which invokes unbind_workers(cpu), the interesting bit being:

                for_each_pool_worker(worker, pool) {
                        kthread_set_per_cpu(worker->task, -1);
                        WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, cpu_possible_mask) < 0);
                }

The rcu_torture_reader() kthreads aren't bound to any particular CPU are
they? I can't find any code that would indicate they are - and in that case
it means we have a problem with is_cpu_allowed() or related.