Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1752359AbdG2W27 (ORCPT <rfc822;w@1wt.eu>);
        Sat, 29 Jul 2017 18:28:59 -0400
Received: from mail-oi0-f42.google.com ([209.85.218.42]:36593 "EHLO
        mail-oi0-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1751634AbdG2W25 (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Sat, 29 Jul 2017 18:28:57 -0400
MIME-Version: 1.0
In-Reply-To: <CAJWu+ooaqWVTad8JNc4gnCAjkgZj4toCb7RrCKNQfm=x3tD1Gw@mail.gmail.com>
References: <CAJWu+oo0_zAWLjObRZp-r3VH5kKKb4aFVnRevtbYh+uH_tCNag@mail.gmail.com>
 <20170630004912.GA2457@destiny> <CAJWu+opFTvnGnYyRo9eYZJUxiHdoMCMoadUvnX3p=6ypwWX_Jw@mail.gmail.com>
 <20170630142815.GA9743@destiny> <1498842140.15161.66.camel@gmail.com>
 <CAJWu+oo7AY+PYqdwSoW6ToRe2o40MhT1wSCrRU8nkQuH0cFO4w@mail.gmail.com>
 <1501340845.7706.168.camel@gmail.com> <CAJWu+ooaqWVTad8JNc4gnCAjkgZj4toCb7RrCKNQfm=x3tD1Gw@mail.gmail.com>
From: Joel Fernandes <joelaf@google.com>
Date: Sat, 29 Jul 2017 15:28:56 -0700
Message-ID: <CAJWu+ore7pfzMw4dz=y4YMi5YGs_b5DwjA6=GfXFsqqJN10gWQ@mail.gmail.com>
Subject: Re: wake_wide mechanism clarification
To: Mike Galbraith <umgwanakikbuti@gmail.com>
Cc: Josef Bacik <josef@toxicpanda.com>,
        Peter Zijlstra <peterz@infradead.org>,
        LKML <linux-kernel@vger.kernel.org>, Juri Lelli <Juri.Lelli@arm.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Patrick Bellasi <patrick.bellasi@arm.com>,
        Brendan Jackman <brendan.jackman@arm.com>,
        Chris Redpath <Chris.Redpath@arm.com>,
        Michael Wang <wangyun@linux.vnet.ibm.com>,
        Matt Fleming <matt@codeblueprint.co.uk>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 2892
Lines: 62

Hi Mike,

On Sat, Jul 29, 2017 at 1:19 PM, Joel Fernandes <joelaf@google.com> wrote:
<snip>
>
>>> To explain the second condition above, Michael Wang said the following in [1]
>>>
>>> "Furthermore, if waker also has a high 'nr_wakee_switch', imply that multiple
>>> tasks rely on it, then waker's higher latency will damage all of them, pull
>>> wakee seems to be a bad deal."
>>
>> Yes, "Furthermore". To detect 1:N, Michael chose llc_size as his N.  Is
>> the one flipping partners at least N/s, and the other about N times as
>> often?  If so, the two may be part of a too big to wisely pull 1:N.
>>
>> If you have a better idea, by all means, pull it out.  Nobody is
>
> Sure yeah, first I'm trying to understand the heuristic itself which
> I'm glad to be making progress with thanks to yours and others' help!
>
>> attached to wake_wide(), in fact, I suspect Peter hates it.  I'm not
>> fond of it either, it having obvious holes.  The only thing it has
>> going for it is simplicity.  Bend it up, replace it, fire away.
>>
>
> Ok, it makes much more sense to me now. Also for the N:N case,
> wouldn't the excessive wake-affine increase the latency and a
> spreading might be better? Say if slave and master flips are much
> greater than factor (llc_size), then slave > factor && master < slave
> * factor, would probably return true a lot (and we would return 0
> causing an affine wakeup). That's probably a bad thing right as it
> could overload the waker's CPU quickly? I guess the heuristic tries to
> maximize cache-hits more than reduce latency?
>
>>> Again I didn't follow why the second condition couldn't just be:
>>> waker->nr_wakee_switch > factor, or, (waker->nr_wakee_switch +
>>> wakee->nr_wakee_switch) > factor, based on the above explanation from
>>> Micheal Wang that I quoted.
>>> and why he's instead doing the whole multiplication thing there that I
>>> was talking about earlier: "factor * wakee->nr_wakee_switch".
>>>
>>> Rephrasing my question in another way, why are we talking the ratio of
>>> master/slave instead of the sum when comparing if its > factor? I am
>>> surely missing something here.
>>
>> Because the heuristic tries to not demolish 1:1 buddies.  Big partner
>> flip delta means the pair are unlikely to be a communicating pair,
>> perhaps at high frequency where misses hurt like hell.
>
> But it does seem to me to demolish the N:N communicating pairs from a
> latency/load balancing standpoint. For he case of N readers and N
> writers, the ratio (master/slave) comes down to 1:1 and we wake
> affine. Hopefully I didn't miss something too obvious about that.

I think wake_affine() should correctly handle the case (of
overloading) I bring up here where wake_wide() is too conservative and
does affine a lot, (I don't have any data for this though, this just
from code reading), so I take this comment back for this reason.

thanks,

-Joel