Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [PATCH] sched/fair: Optimize select_idle_cpu
To:     <mingo@kernel.org>, <peterz@infradead.org>,
        <linux-kernel@vger.kernel.org>
CC:     <chenwandun@huawei.com>, <xiexiuqi@huawei.com>,
        <liwei391@huawei.com>, <huawei.libin@huawei.com>,
        <bobo.shaobowang@huawei.com>, <juri.lelli@redhat.com>,
        <vincent.guittot@linaro.org>,
        "chengjian (D)" <cj.chengjian@huawei.com>
References: <20191212144102.181510-1-cj.chengjian@huawei.com>
From:   "chengjian (D)" <cj.chengjian@huawei.com>
Message-ID: <3cf0931f-d2f4-008e-8e58-cbc9bd45e3a6@huawei.com>
Date:   Thu, 12 Dec 2019 22:56:38 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101
 Thunderbird/68.3.0
MIME-Version: 1.0
In-Reply-To: <20191212144102.181510-1-cj.chengjian@huawei.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


On 2019/12/12 22:41, Cheng Jian wrote:
> Our threads are all bind to the front CPUs of the LLC domain,
> and now all the threads runs on the last CPU of them. nr is
> always less than the cpumask_weight, for_each_cpu_wrap can't
> find the CPU which our threads can run on, so the threads stay
> at the last CPU all the time.
Test :

Run on ARM64 4NODE, 128 CORE

// cat a.c
#include <time.h>

int main(void)
{
     struct timespec time, save;
     int ret;

     time.tv_sec = 0;
     time.tv_nsec = 1;

     while (1) {
         ret = nanosleep(&time, &save);
         if (ret)
             nanosleep(&save, &save);
     }
     return 0;
}

#cat a.sh
for i in `seq 0 9`
do
     taskset -c 8-11 ./a.out &
done


then run:

     gcc a.c -o a.out
     sh a.sh


without this patch, you can see all the task run on CPU11 all the times.


     %Cpu8  :  0.0 us,  0.0 sy,  0.0 ni, 98.4 id,  0.0 wa,  1.6 hi, 0.0 
si,  0.0 st
     %Cpu9  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi, 0.0 
si,  0.0 st
     %Cpu10 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi, 0.0 
si,  0.0 st
     %Cpu11 :  5.7 us, 40.0 sy,  0.0 ni, 45.7 id,  0.0 wa,  8.6 hi, 0.0 
si,  0.0 st