Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S965106AbcCIXT6 (ORCPT ); Wed, 9 Mar 2016 18:19:58 -0500 Received: from youngberry.canonical.com ([91.189.89.112]:40981 "EHLO youngberry.canonical.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934459AbcCIXRo (ORCPT ); Wed, 9 Mar 2016 18:17:44 -0500 From: Kamal Mostafa To: linux-kernel@vger.kernel.org, stable@vger.kernel.org, kernel-team@lists.ubuntu.com Cc: Tejun Heo , Tang Chen , "Rafael J . Wysocki" , Len Brown , Luis Henriques , Kamal Mostafa Subject: [PATCH 3.13.y-ckt 077/138] workqueue: handle NUMA_NO_NODE for unbound pool_workqueue lookup Date: Wed, 9 Mar 2016 15:13:24 -0800 Message-Id: <1457565265-15195-78-git-send-email-kamal@canonical.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1457565265-15195-1-git-send-email-kamal@canonical.com> References: <1457565265-15195-1-git-send-email-kamal@canonical.com> X-Extended-Stable: 3.13 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2700 Lines: 68 3.13.11-ckt36 -stable review patch. If anyone has any objections, please let me know. ---8<------------------------------------------------------------ From: Tejun Heo commit d6e022f1d207a161cd88e08ef0371554680ffc46 upstream. When looking up the pool_workqueue to use for an unbound workqueue, workqueue assumes that the target CPU is always bound to a valid NUMA node. However, currently, when a CPU goes offline, the mapping is destroyed and cpu_to_node() returns NUMA_NO_NODE. This has always been broken but hasn't triggered often enough before 874bbfe600a6 ("workqueue: make sure delayed work run in local cpu"). After the commit, workqueue forcifully assigns the local CPU for delayed work items without explicit target CPU to fix a different issue. This widens the window where CPU can go offline while a delayed work item is pending causing delayed work items dispatched with target CPU set to an already offlined CPU. The resulting NUMA_NO_NODE mapping makes workqueue try to queue the work item on a NULL pool_workqueue and thus crash. While 874bbfe600a6 has been reverted for a different reason making the bug less visible again, it can still happen. Fix it by mapping NUMA_NO_NODE to the default pool_workqueue from unbound_pwq_by_node(). This is a temporary workaround. The long term solution is keeping CPU -> NODE mapping stable across CPU off/online cycles which is being worked on. Signed-off-by: Tejun Heo Reported-by: Mike Galbraith Cc: Tang Chen Cc: Rafael J. Wysocki Cc: Len Brown Link: http://lkml.kernel.org/g/1454424264.11183.46.camel@gmail.com Link: http://lkml.kernel.org/g/1453702100-2597-1-git-send-email-tangchen@cn.fujitsu.com [ luis: backported to 3.16: adjusted context ] Signed-off-by: Luis Henriques Signed-off-by: Kamal Mostafa --- kernel/workqueue.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index eb1cc22..de5ad5c 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -558,6 +558,16 @@ static struct pool_workqueue *unbound_pwq_by_node(struct workqueue_struct *wq, int node) { assert_rcu_or_wq_mutex(wq); + + /* + * XXX: @node can be NUMA_NO_NODE if CPU goes offline while a + * delayed item is pending. The plan is to keep CPU -> NODE + * mapping valid and stable across CPU on/offlines. Once that + * happens, this workaround can be removed. + */ + if (unlikely(node == NUMA_NO_NODE)) + return wq->dfl_pwq; + return rcu_dereference_raw(wq->numa_pwq_tbl[node]); } -- 2.7.0