Message-ID: <551D8F20.8020107@ezchip.com>
Date: Thu, 2 Apr 2015 14:49:04 -0400
From: Chris Metcalf <cmetcalf@ezchip.com>
User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0
MIME-Version: 1.0
To: Peter Zijlstra <peterz@infradead.org>
CC: Frederic Weisbecker <fweisbec@gmail.com>, Don Zickus <dzickus@redhat.com>,
        Ingo Molnar <mingo@kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Andrew Jones <drjones@redhat.com>,
        chai wen <chaiw.fnst@cn.fujitsu.com>,
        Ulrich Obergfell <uobergfe@redhat.com>,
        Fabian Frederick <fabf@skynet.be>, Aaron Tomlin <atomlin@redhat.com>,
        Ben Zhang <benzh@chromium.org>, Christoph Lameter <cl@linux.com>,
        Gilad Ben-Yossef <gilad@benyossef.com>,
        Steven Rostedt <rostedt@goodmis.org>, <linux-kernel@vger.kernel.org>,
        Jonathan Corbet <corbet@lwn.net>, <linux-doc@vger.kernel.org>
Subject: Re: [PATCH v3] watchdog: add watchdog_cpumask sysctl to assist nohz
References: <551AE7D4.3020608@ezchip.com> <20150402133502.GA175361@redhat.com> <551D48F9.6090101@ezchip.com> <20150402141527.GD175361@redhat.com> <20150402153827.GC10357@lerouge> <551D6373.2030000@ezchip.com> <20150402164845.GD10357@lerouge> <1427996368-2199-1-git-send-email-cmetcalf@ezchip.com> <20150402180626.GG23123@twins.programming.kicks-ass.net> <551D8769.5030100@ezchip.com> <20150402183308.GB27490@worktop.programming.kicks-ass.net>
In-Reply-To: <20150402183308.GB27490@worktop.programming.kicks-ass.net>
Content-Type: text/plain; charset="windows-1252"; format=flowed
Content-Transfer-Encoding: 7bit
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Apr 2015 18:49:15.1201 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM2PR02MB0772
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 4435
Lines: 118

On 04/02/2015 02:33 PM, Peter Zijlstra wrote:
> On Thu, Apr 02, 2015 at 02:16:09PM -0400, Chris Metcalf wrote:
>> On 04/02/2015 02:06 PM, Peter Zijlstra wrote:
>>> On Thu, Apr 02, 2015 at 01:39:28PM -0400, cmetcalf@ezchip.com wrote:
>>>> @@ -431,6 +434,10 @@ static void watchdog_enable(unsigned int cpu)
>>>>   	hrtimer_init(hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
>>>>   	hrtimer->function = watchdog_timer_fn;
>>>> +	/* Exit if the cpu is not allowed for watchdog. */
>>>> +	if (!cpumask_test_cpu(cpu, watchdog_mask))
>>>> +		do_exit(0);
>>>> +
>>> Ick, that doesn't look right for smpboot threads.
>> I didn't see a better way to make this happen without adding
>> a bunch of infrastructure to the smpboot thread mechanism
>> to use a cpumask other than for_each_online_cpu().  The exit
>> seems benign in my testing, but I agree it's not the cleanest
>> way to express what we're trying to do here.
>>
>> Perhaps something like an optional cpumask_t pointer in
>> struct smp_hotplug_thread, which if present specifies the
>> cpus to run on, and otherwise we stick with cpu_online_mask?
> What's wrong with just leaving the thread be but making sure it'll never
> actually do anything?

I think a common case for nohz_full systems is that you'll
have a whole lot of watchdog threads that never do anything.
Our TILEGx-72 systems are often run with one housekeeping
core and the rest doing userspace nohz_full driver work.  So
not creating the threads seems tidier - it keeps 71 threads out
of the "ps" listing :-)

Here's a quick sketch of the delta from my previous patch to
one with a new smp_hotplug_thread.cpumask field.  If folks
are OK with modifying the smpboot threads like this, I think
it probably is a cleaner approach:

diff --git a/include/linux/smpboot.h b/include/linux/smpboot.h
index 13e929679550..f28519612ee3 100644
--- a/include/linux/smpboot.h
+++ b/include/linux/smpboot.h
@@ -27,6 +27,7 @@ struct smpboot_thread_data;
   * @pre_unpark:		Optional unpark function, called before the thread is
   *			unparked (cpu online). This is not guaranteed to be
   *			called on the target cpu of the thread. Careful!
+ * @cpumask:		Optional cpumask, specifying what cores to run on.
   * @selfparking:	Thread is not parked by the park function.
   * @thread_comm:	The base name of the thread
   */
@@ -41,6 +42,7 @@ struct smp_hotplug_thread {
  	void				(*park)(unsigned int cpu);
  	void				(*unpark)(unsigned int cpu);
  	void				(*pre_unpark)(unsigned int cpu);
+	cpumask_t			*cpumask;
  	bool				selfparking;
  	const char			*thread_comm;
  };
diff --git a/kernel/smpboot.c b/kernel/smpboot.c
index 40190f28db35..be503c2ddb5f 100644
--- a/kernel/smpboot.c
+++ b/kernel/smpboot.c
@@ -172,6 +172,9 @@ __smpboot_create_thread(struct smp_hotplug_thread *ht, unsigned int cpu)
  	if (tsk)
  		return 0;
  
+	if (ht->cpumask && !cpumask_test_cpu(cpu, ht->cpumask))
+		return 0;
+
  	td = kzalloc_node(sizeof(*td), GFP_KERNEL, cpu_to_node(cpu));
  	if (!td)
  		return -ENOMEM;
@@ -220,9 +223,11 @@ static void smpboot_unpark_thread(struct smp_hotplug_thread *ht, unsigned int cp
  {
  	struct task_struct *tsk = *per_cpu_ptr(ht->store, cpu);
  
-	if (ht->pre_unpark)
-		ht->pre_unpark(cpu);
-	kthread_unpark(tsk);
+	if (tsk) {
+		if (ht->pre_unpark)
+			ht->pre_unpark(cpu);
+		kthread_unpark(tsk);
+	}
  }
  
  void smpboot_unpark_threads(unsigned int cpu)
diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index 2140c2d81dc9..681e5648e093 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -434,10 +434,6 @@ static void watchdog_enable(unsigned int cpu)
  	hrtimer_init(hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
  	hrtimer->function = watchdog_timer_fn;
  
-	/* Exit if the cpu is not allowed for watchdog. */
-	if (!cpumask_test_cpu(cpu, watchdog_mask))
-		do_exit(0);
-
  	/* Enable the perf event */
  	watchdog_nmi_enable(cpu);
  
@@ -588,6 +584,7 @@ static struct smp_hotplug_thread watchdog_threads = {
  	.cleanup		= watchdog_cleanup,
  	.park			= watchdog_disable,
  	.unpark			= watchdog_enable,
+	.cpumask		= watchdog_mask,
  };
  
  static void restart_watchdog_hrtimer(void *info)

-- 
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/