Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [RFC PATCH 2/3] sched: change scheduler to give preference to
 soft affinity CPUs
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     linux-kernel@vger.kernel.org, mingo@redhat.com, tglx@linutronix.de,
        prakash.sangappa@oracle.com, dhaval.giani@oracle.com,
        daniel.lezcano@linaro.org, vincent.guittot@linaro.org,
        viresh.kumar@linaro.org, tim.c.chen@linux.intel.com,
        mgorman@techsingularity.net, Paul Turner <pjt@google.com>
References: <20190626224718.21973-1-subhra.mazumdar@oracle.com>
 <20190626224718.21973-3-subhra.mazumdar@oracle.com>
 <20190702172851.GA3436@hirez.programming.kicks-ass.net>
 <a91c09ce-aec1-eaa1-4daf-70024cebf360@oracle.com>
 <20190718113758.GN3402@hirez.programming.kicks-ass.net>
From:   Subhra Mazumdar <subhra.mazumdar@oracle.com>
Message-ID: <067f898a-3d3b-0ff5-724e-50ed2e989286@oracle.com>
Date:   Fri, 19 Jul 2019 08:25:01 +0530
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0)
 Gecko/20100101 Thunderbird/60.8.0
MIME-Version: 1.0
In-Reply-To: <20190718113758.GN3402@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


On 7/18/19 5:07 PM, Peter Zijlstra wrote:
> On Wed, Jul 17, 2019 at 08:31:25AM +0530, Subhra Mazumdar wrote:
>> On 7/2/19 10:58 PM, Peter Zijlstra wrote:
>>> On Wed, Jun 26, 2019 at 03:47:17PM -0700, subhra mazumdar wrote:
>>>> The soft affinity CPUs present in the cpumask cpus_preferred is used by the
>>>> scheduler in two levels of search. First is in determining wake affine
>>>> which choses the LLC domain and secondly while searching for idle CPUs in
>>>> LLC domain. In the first level it uses cpus_preferred to prune out the
>>>> search space. In the second level it first searches the cpus_preferred and
>>>> then cpus_allowed. Using affinity_unequal flag it breaks early to avoid
>>>> any overhead in the scheduler fast path when soft affinity is not used.
>>>> This only changes the wake up path of the scheduler, the idle balancing
>>>> is unchanged; together they achieve the "softness" of scheduling.
>>> I really dislike this implementation.
>>>
>>> I thought the idea was to remain work conserving (in so far as that
>>> we're that anyway), so changing select_idle_sibling() doesn't make sense
>>> to me. If there is idle, we use it.
>>>
>>> Same for newidle; which you already retained.
>> The scheduler is already not work conserving in many ways. Soft affinity is
>> only for those who want to use it and has no side effects when not used.
>> Also the way scheduler is implemented in the first level of search it may
>> not be possible to do it in a work conserving way, I am open to ideas.
> I really don't understand the premise of this soft affinity stuff then.
>
> I understood it was to allow spreading if under-utilized, but group when
> over-utilized, but you're arguing for the exact opposite, which doesn't
> make sense.
You are right on the premise. The whole knob thing came into existence
because I couldn't make the first level of search work conserving. I am
concerned that trying to make that work conserving can introduce
significant latency in the code path when SA is used. I have made the
second level of search work conserving when we search the LLC domain.

Having said that, SA need not necessarily be binary i.e only spill over to
the allowed set if the preferred set is 100% utilized (work conserving).
The spill over can happen before that and SA can have a degree of softness.

The above two points made me go down the knob path for the first level of
search.