Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753525AbdIHCNa (ORCPT ); Thu, 7 Sep 2017 22:13:30 -0400 Received: from smtp.codeaurora.org ([198.145.29.96]:58970 "EHLO smtp.codeaurora.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752923AbdIHCN1 (ORCPT ); Thu, 7 Sep 2017 22:13:27 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 990F0602B3 Authentication-Results: pdx-caf-mail.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: pdx-caf-mail.web.codeaurora.org; spf=none smtp.mailfrom=prsood@codeaurora.org Subject: Re: [PATCH] cgroup/cpuset: remove circular dependency deadlock To: Peter Zijlstra References: <1504792583-10424-1-git-send-email-prsood@codeaurora.org> <20170907174505.GF17526@worktop.programming.kicks-ass.net> Cc: tj@kernel.org, lizefan@huawei.com, cgroups@vger.kernel.org, mingo@kernel.org, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, linux-kernel@vger.kernel.org, sramana@codeaurora.org From: Prateek Sood Message-ID: Date: Fri, 8 Sep 2017 07:43:21 +0530 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <20170907174505.GF17526@worktop.programming.kicks-ass.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2398 Lines: 75 On 09/07/2017 11:15 PM, Peter Zijlstra wrote: > On Thu, Sep 07, 2017 at 07:26:23PM +0530, Prateek Sood wrote: >> Remove circular dependency deadlock in a scenario where hotplug of CPU is >> being done while there is updation in cgroup and cpuset triggered from >> userspace. >> >> Process A => kthreadd => Process B => Process C => Process A > >> Process A >> cpu_subsys_offline(); >> cpu_down(); >> _cpu_down(); >> percpu_down_write(&cpu_hotplug_lock); //held >> cpuhp_invoke_callback(); >> workqueue_offline_cpu(); >> wq_update_unbound_numa(); >> kthread_create_on_node(); >> wake_up_process(); //wakeup kthreadd > > TJ, I'm puzzled, why would we need to spawn new threads to update NUMA > affinity when taking a CPU out? That doesn't make sense to me, we can > either shrink the affinity of an existing thread or completely kill of a > thread if the mask becomes empty. But why spawn a new thread? > >> flush_work(); >> wait_for_completion(); >> > > Yes, inverting cpuset and hotplug would break that chain, but I'm still > wondering why workqueue needs to spawn threads on CPU down. > Thanks for the comments Peter You rightly mentioned that a new thread will not be spawn while updating NUMA affinity when taking a CPU out. While a CPU is made offline, attempt is made to unbind per-cpu worker for CPU going down. This is done by queuing unbind work to system_highpri_wq. It results in an attempt to create one bounded worker thread as there is none. wait_for_completion() in flush_work() waits for unbinding to finish for CPU going down. Process A cpu_subsys_offline(); cpu_down(); _cpu_down(); percpu_down_write(&cpu_hotplug_lock); //held cpuhp_invoke_callback(); workqueue_offline_cpu(); queue_work_on(system_highpri_wq); __queue_work(); insert_work(); wake_up_worker(); //pool->nr_running = 0 flush_work(); wait_for_completion(); worker_thread(); need_more_worker(); // returns true manage_workers(); maybe_create_worker(); create_worker(); kthread_create_on_node(); wake_up_process(kthreadd_task); -- Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center, Inc., is a member of Code Aurora Forum, a Linux Foundation Collaborative Project