Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [RFC PATCH v2 11/17] sched: Basic tracking of matching tasks
To:     Aubrey Li <aubrey.intel@gmail.com>
Cc:     Tim Chen <tim.c.chen@linux.intel.com>,
        Aaron Lu <aaron.lu@linux.alibaba.com>,
        Vineeth Remanan Pillai <vpillai@digitalocean.com>,
        Nishanth Aravamudan <naravamudan@digitalocean.com>,
        Julien Desfossez <jdesfossez@digitalocean.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Paul Turner <pjt@google.com>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Linux List Kernel Mailing <linux-kernel@vger.kernel.org>,
        =?UTF-8?B?RnLDqWTDqXJpYyBXZWlzYmVja2Vy?= <fweisbec@gmail.com>,
        Kees Cook <keescook@chromium.org>,
        Greg Kerr <kerrnel@google.com>, Phil Auld <pauld@redhat.com>,
        Aaron Lu <aaron.lwe@gmail.com>,
        Valentin Schneider <valentin.schneider@arm.com>,
        Mel Gorman <mgorman@techsingularity.net>,
        Pawan Gupta <pawan.kumar.gupta@linux.intel.com>,
        Paolo Bonzini <pbonzini@redhat.com>
References: <cover.1556025155.git.vpillai@digitalocean.com>
 <2364f2b65bf50826d881c84d7634b6565dfee527.1556025155.git.vpillai@digitalocean.com>
 <20190429061516.GA9796@aaronlu>
 <6dfc392f-e24b-e641-2f7d-f336a90415fa@linux.intel.com>
 <777b7674-4811-dac4-17df-29bd028d6b26@linux.intel.com>
 <CAERHkrvU0nay-cG9equdOBejOZ5Ffdxo+67ZRp9q0L9BQkcAtQ@mail.gmail.com>
 <eb9abb34-d946-c63c-750b-8f52ed842670@oracle.com>
 <28fb6854-2772-5d29-087a-6a0cf6afe626@oracle.com>
 <CAERHkrsavsBoEOR5Eq-nm6ADarS0zTi5Mu-T7TO6JoSUi7TRfQ@mail.gmail.com>
From:   Subhra Mazumdar <subhra.mazumdar@oracle.com>
Message-ID: <8098b70b-2095-91ea-d4ad-9181829066c7@oracle.com>
Date:   Wed, 8 May 2019 17:25:18 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.5.1
MIME-Version: 1.0
In-Reply-To: <CAERHkrsavsBoEOR5Eq-nm6ADarS0zTi5Mu-T7TO6JoSUi7TRfQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


On 5/8/19 5:01 PM, Aubrey Li wrote:
> On Thu, May 9, 2019 at 2:41 AM Subhra Mazumdar
> <subhra.mazumdar@oracle.com> wrote:
>>
>> On 5/8/19 11:19 AM, Subhra Mazumdar wrote:
>>> On 5/8/19 8:49 AM, Aubrey Li wrote:
>>>>> Pawan ran an experiment setting up 2 VMs, with one VM doing a
>>>>> parallel kernel build and one VM doing sysbench,
>>>>> limiting both VMs to run on 16 cpu threads (8 physical cores), with
>>>>> 8 vcpu for each VM.
>>>>> Making the fix did improve kernel build time by 7%.
>>>> I'm gonna agree with the patch below, but just wonder if the testing
>>>> result is consistent,
>>>> as I didn't see any improvement in my testing environment.
>>>>
>>>> IIUC, from the code behavior, especially for 2 VMs case(only 2
>>>> different cookies), the
>>>> per-rq rb tree unlikely has nodes with different cookies, that is, all
>>>> the nodes on this
>>>> tree should have the same cookie, so:
>>>> - if the parameter cookie is equal to the rb tree cookie, we meet a
>>>> match and go the
>>>> third branch
>>>> - else, no matter we go left or right, we can't find a match, and
>>>> we'll return idle thread
>>>> finally.
>>>>
>>>> Please correct me if I was wrong.
>>>>
>>>> Thanks,
>>>> -Aubrey
>>> This is searching in the per core rb tree (rq->core_tree) which can have
>>> 2 different cookies. But having said that, even I didn't see any
>>> improvement with the patch for my DB test case. But logically it is
>>> correct.
>>>
>> Ah, my bad. It is per rq. But still can have 2 different cookies. Not sure
>> why you think it is unlikely?
> Yeah, I meant 2 different cookies on the system, but unlikely 2
> different cookies
> on one same rq.
>
> If I read the source correctly, for the sched_core_balance path, when try to
> steal cookie from another CPU, sched_core_find() uses dst's cookie to search
> if there is a cookie match in src's rq, and sched_core_find() returns idle or
> matched task, and later put this matched task onto dst's rq (activate_task() in
> sched_core_find()). At this moment, the nodes on the rq's rb tree should have
> same cookies.
>
> Thanks,
> -Aubrey
Yes, but sched_core_find is also called from pick_task to find a local
matching task. The enqueue side logic of the scheduler is unchanged with
core scheduling, so it is possible tasks with different cookies are
enqueued on the same rq. So while searching for a matching task locally
doing it correctly should matter.