Message-ID: <5564CEDB.9000700@ezchip.com>
Date: Tue, 26 May 2015 15:51:55 -0400
From: Chris Metcalf <cmetcalf@ezchip.com>
User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0
MIME-Version: 1.0
To: Mike Galbraith <umgwanakikbuti@gmail.com>
CC: Frederic Weisbecker <fweisbec@gmail.com>,
        Steven Rostedt <rostedt@goodmis.org>, Ingo Molnar <mingo@kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Gilad Ben Yossef <giladb@ezchip.com>, Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>, Rik van Riel <riel@redhat.com>,
        Tejun Heo <tj@kernel.org>, Thomas Gleixner <tglx@linutronix.de>,
        "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
        Christoph Lameter <cl@linux.com>, <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 0/6] support "dataplane" mode for nohz_full
References: <1431107927-13998-1-git-send-email-cmetcalf@ezchip.com>	 <20150508141824.797eb0d89d514e39fd30fffe@linux-foundation.org>	 <20150508172210.559830a9@gandalf.local.home> <554D428E.6020702@ezchip.com>	 <20150508161909.308d60e21f6b83b897174276@linux-foundation.org>	 <20150509070538.GA9413@gmail.com>	 <20150511085759.71deeb64@gandalf.local.home>	 <20150511153602.GA32512@lerouge> <1431371974.3195.126.camel@gmail.com>	 <55510218.9090104@ezchip.com> <1431395275.3195.19.camel@gmail.com>	 <55560B3A.6000109@ezchip.com> <1431715453.3170.53.camel@gmail.com>
In-Reply-To: <1431715453.3170.53.camel@gmail.com>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 May 2015 19:52:03.0557 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM2PR02MB0770
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 2739
Lines: 55

Thanks for the clarification, and sorry for the slow reply; I had a busy
week of meetings last week, and then the long weekend in the U.S.

On 05/15/2015 02:44 PM, Mike Galbraith wrote:
> Just because the nohz_full feature itself is currently static is no
> reason to put users thereof in a straight jacket by mandating that any
> set they define irrevocably disappears from the generic resource pool .
> Those CPUS are useful until the moment someone cripples them, which
> making nohz_full imply isolcpus does if isolcpus then also becomes
> immutable, which Rik's patch does.  Making nohz_full imply isolcpus
> sounds perfectly fine until someone comes along and makes isolcpus
> immutable (Rik's patch), at which point the user loses a choice due to
> two people making it imply things that _alone_ sound perfectly fine.
>
> See what I'm saying now?

That does make sense; my argument was that 99% of the time when
someone specifies nohz_full they also need isolcpus.  You're right
that someone playing with nohz_full would be unpleasantly surprised.
And of course having more flexibility always feels like a plus.
On balance I suspect it's still better to make command line arguments
handle the common cases most succinctly.

Hopefully we'll get a to a point where all of this is dynamic and how
we play with the boot arguments no longer matters.  If not, perhaps
we revisit this and make a cpu_isolation=1-15 type command line
argument that enables isolcpus and nohz_full both.

>>> Thomas has nuked the hrtimer softirq.
>> Yes, this I didn't know.  So I will drop my "no ksoftirqd" patch and
>> we will see if ksoftirqs emerge as an issue for my "cpu isolation"
>> stuff in the future; it may be that that was the only issue.
>>
>>> Inlining softirqs may save a context switch, but adds cycles that we may
>>> consume at higher frequency than the thing we're avoiding.
>> Yes but consuming cycles is not nearly as much of a concern
>> as avoiding interrupts or scheduling, certainly for the case of
>> userspace drivers that I described above.
> If you're raising softirqs in an SMP kernel, you're also doing something
> that puts you at very serious risk of meeting the jitter monster, locks,
> and worse, sleeping locks, no?

The softirqs were being raised by third parties for hrtimer, not by
the application code itself, if I remember correctly.  In any case
this appears not to be an issue for nohz_full any more now.

-- 
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/