Message-ID: <5527ED3E.1060101@ezchip.com>
Date: Fri, 10 Apr 2015 11:33:18 -0400
From: Chris Metcalf <cmetcalf@ezchip.com>
User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0
MIME-Version: 1.0
To: Mike Galbraith <umgwanakikbuti@gmail.com>,
        Peter Zijlstra <peterz@infradead.org>
CC: Frederic Weisbecker <fweisbec@gmail.com>,
        "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
        "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>,
        Martin Schwidefsky <schwidefsky@de.ibm.com>,
        Ingo Molnar <mingo@redhat.com>, <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v5] nohz: set isolcpus when nohz_full is set
References: <20150409124524.GA17709@lerouge>	 <1428598779-24244-1-git-send-email-cmetcalf@ezchip.com>	 <20150409171251.GS21418@twins.programming.kicks-ass.net> <1428627910.3203.4.camel@gmail.com>
In-Reply-To: <1428627910.3203.4.camel@gmail.com>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Apr 2015 15:33:26.8879 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5PR02MB0773
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 3094
Lines: 75

On 04/09/2015 09:05 PM, Mike Galbraith wrote:
> On Thu, 2015-04-09 at 19:12 +0200, Peter Zijlstra wrote:
>> On Thu, Apr 09, 2015 at 12:59:39PM -0400, Chris Metcalf wrote:
>>> diff --git a/include/linux/sched.h b/include/linux/sched.h
>>> index 6d77432e14ff..18a961b9beba 100644
>>> --- a/include/linux/sched.h
>>> +++ b/include/linux/sched.h
>>> @@ -323,6 +323,7 @@ struct task_struct;
>>>   extern int lockdep_tasklist_lock_is_held(void);
>>>   #endif /* #ifdef CONFIG_PROVE_RCU */
>>>   
>>> +extern void sched_isolated_map_add(const struct cpumask *);
>>>   extern void sched_init(void);
>>>   extern void sched_init_smp(void);
>>>   extern asmlinkage void schedule_tail(struct task_struct *prev);
>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>>> index f0f831e8a345..b055c5e0e65c 100644
>>> --- a/kernel/sched/core.c
>>> +++ b/kernel/sched/core.c
>>> @@ -5824,6 +5824,11 @@ static int __init isolated_cpu_setup(char
>>> *str)
>>>   
>>>   __setup("isolcpus=", isolated_cpu_setup);
>>>   
>>> +void sched_isolated_map_add(const struct cpumask *cpumask)
>>> +{
>>> +   cpumask_or(cpu_isolated_map, cpu_isolated_map, cpumask);
>>> +}
>>> +
>>>   struct s_data {
>>>      struct sched_domain ** __percpu sd;
>>>      struct root_domain      *rd;
>>> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
>>> index a4c4edac4528..b0092d02ca3f 100644
>>> --- a/kernel/time/tick-sched.c
>>> +++ b/kernel/time/tick-sched.c
>>> @@ -385,6 +385,9 @@ void __init tick_nohz_init(void)
>>>      for_each_cpu(cpu, tick_nohz_full_mask)
>>>              context_tracking_cpu_set(cpu);
>>>   
>>> +   /* It's not meaningful to be nohz without disabling the
>>> scheduler. */
>>> +   sched_isolated_map_add(tick_nohz_full_mask);
>>> +
>>>      cpu_notifier(tick_nohz_cpu_down_callback, 0);
>>>      pr_info("NO_HZ: Full dynticks CPUs: %*pbl.\n",
>>>              cpumask_pr_args(tick_nohz_full_mask));
>> Right, this could work. Although I would suggest adding a comment
>> somewhere that we should be careful with init order. I checked, this
>> appears to be ordered right, but...
> I'd embed it in domain construction, that way it'd be ready for the
> day nohz_full becomes dynamic, and people can start using cpusets to
> set up /tear down isolated sets on the fly.

So, move it to the top of build_sched_domains(), and then for
every "const struct cpumask *cpu_map" argument, create a
temporary cpu_map so we can mask out the nohz_full cores?

The problem is, we already allow partition_sched_domains()
to override "isolcpus=", so it seems appropriate that you should
be able to override "nohz_full=" in the same way, which my
current patch (v6) does.

So I think the proposed solution is certainly no worse than what
we have now in terms of a future migration to cpusets.

-- 
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/