Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Tue, 14 Jan 2020 12:48:12 -0500
From:   Steven Rostedt <rostedt@goodmis.org>
To:     David Laight <David.Laight@ACULAB.COM>
Cc:     'Vincent Guittot' <vincent.guittot@linaro.org>,
        Peter Zijlstra <peterz@infradead.org>,
        Viresh Kumar <viresh.kumar@linaro.org>,
        Ingo Molnar <mingo@redhat.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        linux-kernel <linux-kernel@vger.kernel.org>
Subject: Re: sched/fair: scheduler not running high priority process on idle
 cpu
Message-ID: <20200114124812.4d5355ae@gandalf.local.home>
In-Reply-To: <5ba2ae2d426c4058b314c20c25a9b1d0@AcuMS.aculab.com>
References: <212fabd759b0486aa8df588477acf6d0@AcuMS.aculab.com>
        <20200114115906.22f952ff@gandalf.local.home>
        <5ba2ae2d426c4058b314c20c25a9b1d0@AcuMS.aculab.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, 14 Jan 2020 17:33:50 +0000
David Laight <David.Laight@ACULAB.COM> wrote:

> I have added a cond_resched() to the offending loop, but a close look implies
> that code is called with a lock held in another (less common) path so that
> can't be directly committed and so CONFIG_PREEMPT won't help.
> 
> Indeed requiring CONFIG_PREEMPT doesn't help when customers are running
> the application, nor (probably) on AWS since I doubt it is ever the default.
> 
> Does the same apply to non-RT tasks?
> I can select almost any priority, but RT ones are otherwise a lot better.
> 
> I've also seen RT processes delayed by the network stack 'bh' that runs
> in a softint from the hardware interrupt.
> That can take a while (clearing up tx and refilling rx) and I don't think we
> have any control over the cpu it runs on?

Yes, even with CONFIG_PREEMPT, Linux has no guarantees of latency for
any task regardless of priority. If you have latency requirements, then
you need to apply the PREEMPT_RT patch (which may soon make it to
mainline this year!), which spin locks and bh wont stop a task from
scheduling (unless they need the same lock).

> 
> The cost of ftrace function call entry/exit (about 200 clocks) makes it
> rather unsuitable for any performance measurements unless only
> a very few functions are traced - which rather requires you know
> what the code is doing :-(
> 

Well, when I use function tracing, I start all of them, analyze the
trace, then the functions I don't care about (usually spin locks and
other utils), I add to the set_ftrace_notrace file,  which keeps them
from being part of the trace. I keep doing this until I find a set of
functions that doesn't hurt overhead as much and gives me enough
information to know what is happening. It also helps to enable all or
most events (at least scheduling events).

-- Steve