Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Sender: Tejun Heo <htejun@gmail.com>
Date:   Wed, 7 Jun 2023 12:13:27 -1000
From:   Tejun Heo <tj@kernel.org>
To:     K Prateek Nayak <kprateek.nayak@amd.com>
Cc:     Sandeep Dhavale <dhavale@google.com>, jiangshanlai@gmail.com,
        torvalds@linux-foundation.org, peterz@infradead.org,
        linux-kernel@vger.kernel.org, kernel-team@meta.com,
        joshdon@google.com, brho@google.com, briannorris@chromium.org,
        nhuck@google.com, agk@redhat.com, snitzer@kernel.org,
        void@manifault.com, kernel-team@android.com
Subject: Re: [PATCH 14/24] workqueue: Generalize unbound CPU pods
Message-ID: <ZIEBB-A1arYKSK2P@slm.duckdns.org>
References: <20230519001709.2563-1-tj@kernel.org>
 <20230519001709.2563-15-tj@kernel.org>
 <CAB=BE-S=cxewXu7nqJY1DC5w9Bapar_C0cTjpZOQ-Qd5GGwYyw@mail.gmail.com>
 <c307ba94-0d8c-3cbf-19da-44ee31751428@amd.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <c307ba94-0d8c-3cbf-19da-44ee31751428@amd.com>
Precedence: bulk

Hello,

On Wed, May 31, 2023 at 05:44:57PM +0530, K Prateek Nayak wrote:
...
> The RIP points to dereferencing sd_llc_shared->has_idle_cores
> 
>     $ scripts/faddr2line vmlinux select_task_rq_fair+0x9bd
>     select_task_rq_fair+0x9bd/0x2570:
>     test_idle_cores at kernel/sched/fair.c:6830
>     (inlined by) select_idle_sibling at kernel/sched/fair.c:7189
>     (inlined by) select_task_rq_fair at kernel/sched/fair.c:7710

Hmm... the only thing I can think of is workqueue setting ->wake_cpu to
something invalid.

> My kernel is somewhat stable (I have not seen a panic for ~45min but I
> was not stress testing the system either during that time) with the
> following changes:
> 
> diff --git a/kernel/workqueue.c b/kernel/workqueue.c
> index b2e914655f05..a279cc9c2248 100644
> --- a/kernel/workqueue.c
> +++ b/kernel/workqueue.c
> @@ -2247,7 +2247,7 @@ static void unbind_worker(struct worker *worker)
>         if (cpumask_intersects(wq_unbound_cpumask, cpu_active_mask))
>                 WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, wq_unbound_cpumask) < 0);
>         else
> -               WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, cpu_possible_mask) < 0);
> +               WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, cpu_active_mask) < 0);
>  }

I'm not sure why changing the cpus_allowed_ptr would make a difference here.
Maybe the chain of events involves CPUs going offline and the above migrate
the tasks resetting their ->wake_cpu.

Can you please try the following branch and see if any of the warnings
triggers?

 git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git affinity-scopes-dbg-invalid-cpu

Thanks.

-- 
tejun