Date: Wed, 13 Apr 2016 09:44:30 -0400
From: Chris Mason <clm@fb.com>
To: Mike Galbraith <mgalbraith@suse.de>
CC: Peter Zijlstra <peterz@infradead.org>, Ingo Molnar <mingo@kernel.org>,
        Matt Fleming <matt@codeblueprint.co.uk>,
        <linux-kernel@vger.kernel.org>
Subject: Re: sched: tweak select_idle_sibling to look for idle threads
Message-ID: <20160413134430.3s2w4dodocgislpb@floor.thefacebook.com>
Mail-Followup-To: Chris Mason <clm@fb.com>,
	Mike Galbraith <mgalbraith@suse.de>,
	Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@kernel.org>,
	Matt Fleming <matt@codeblueprint.co.uk>,
	linux-kernel@vger.kernel.org
References: <20160409190554.honue3gtian2p6vr@floor.thefacebook.com>
 <1460282661.4251.44.camel@suse.de>
 <20160410195543.fp2tpixaafsts5x3@floor.thefacebook.com>
 <1460350461.3870.36.camel@suse.de>
 <20160412003044.smr24xzuom3locvo@floor.thefacebook.com>
 <1460436248.3839.80.camel@suse.de>
 <20160412132758.7apgqqwl2c2wksy6@floor.thefacebook.com>
 <1460484977.5617.32.camel@suse.de>
 <20160412200728.GA41928@clm-mbp.thefacebook.com>
 <1460517531.3780.29.camel@suse.de>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <1460517531.3780.29.camel@suse.de>
User-Agent: Mutt/1.5.23.1 (2014-03-12)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 1855
Lines: 42

On Wed, Apr 13, 2016 at 05:18:51AM +0200, Mike Galbraith wrote:
> On Tue, 2016-04-12 at 16:07 -0400, Chris Mason wrote:
> 
> > I think that if we're worried about the cost of the idle scan for this
> > workload, find_idlest_group() is either going to hurt much more, or not
> > search enough CPUs to find the idle one.
> 
> find_idlest_group()?  No no no, that's not what I mean at all.
> 
> wake_wide() identifies loads that really want to spread out, thus turns
> off affine wakeups.  We still call select_idle_sibling(), only
> difference being that target is the original cpu, not the waking cpu. 

Ah ok, I see what you mean now.

>  Given making that wide connection bidirectional helped FB's load, it
> seems reasonable that passing wide information to select_idle_sibling()
> would have a good chance of hitting the candidate that stands to gain
> from a full socket scan, while also keeping that cache scrambling scan
> far away from the rest. 
> 
> > But I'm happy to try patches or other ideas, I have a fixed version of
> > the bitmap one going through production benchmarks now.

[ benchmarks say it needs more fixing, ick ]

> 
> Making that wide/full search cheap is still good, because wake_wide()
> also identifies interrupt sources that are waking many, so cheap wide
> search should increase utilization there as well.  The thought was to
> just make the wide thing have a tad wider effect on what it already
> does affect.. and hope that doesn't demolish anything.

So you're interested in numbers where we pass the wake_wide decision
into select_idle_sibling(), and then use that instead of (or in addition
to?) my should_scan_idle() function?

I agree we may need to tweak wake_wide, since most of our wakeups now
are failed affine wakeups.  But, the differences are in p99, so I'll
probably need to get some better metrics.

-chris