Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Tue, 23 Oct 2018 23:06:54 +0530
From:   Srikar Dronamraju <srikar@linux.vnet.ibm.com>
To:     Yi Wang <wang.yi59@zte.com.cn>
Cc:     mingo@redhat.com, peterz@infradead.org,
        linux-kernel@vger.kernel.org, zhong.weidong@zte.com.cn,
        liu.yi24@zte.com.cn
Subject: Re: [PATCH] sched/numa: fix choosing isolated CPUs when
 task_numa_migrate()
Reply-To: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
References: <1540177516-38613-1-git-send-email-wang.yi59@zte.com.cn>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
In-Reply-To: <1540177516-38613-1-git-send-email-wang.yi59@zte.com.cn>
User-Agent: Mutt/1.10.1 (2018-07-13)
Message-Id: <20181023173654.GB7111@linux.vnet.ibm.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


Hi Yi Wang,

> When trying to migrate to a CPU in task_numa_migrate(), we invoke
> task_numa_find_cpu() to choose a spot, in which function we skip
> the CPU which is not in cpus_allowed, but forgot to concern the
> isolated CPUs, and this may cause the task would run on the isolcpus.
>
> This patch fixes this issue by checking the load_balance_mask.
>
> Signed-off-by: Yi Wang <wang.yi59@zte.com.cn>
> Reviewed-by: Yi Liu <liu.yi24@zte.com.cn>
> ---
>  kernel/sched/fair.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
>

I had proposed something similar
http://lkml.kernel.org/r/1491326848-5748-1-git-send-email-srikar@linux.vnet.ibm.com

but at that time, Peter felt we should fix it at different level.
http://lkml.kernel.org/r/20170406073659.y6ubqriyshax4v4m@hirez.programming.kicks-ass.net

and I think he is right.

I have system with 32 cpus, have passed isolcpus as a kernel parameter.

$ grep -o "isolcpus=[,,1-9]*" /proc/cmdline
isolcpus=1,5,9,13
$ grep -i cpus_allowed /proc/$$/status
Cpus_allowed:   ffffdddd
Cpus_allowed_list:      0,2-4,6-8,10-12,14-31


So for a task running on top_cpuset will not have isolcpus as part of the
cpus_allowed.

However if a said task were to call sched_setaffinity, then there is every
likely hood of the cpus being passed being a mix of isolcpus and
nonisolcpus.

For example 
perf bench numa mem --no-data_rand_walk -p 4 -t 8 -G 0 -P 3072 -T 0 -l 50 -c -s 1000

$ for i in $(pgrep -f perf); do  grep -i cpus_allowed_list  /proc/$i/task/*/status ; done | head -n 10
Cpus_allowed_list:      0,2-4,6-8,10-12,14-31
/proc/2107/task/2107/status:Cpus_allowed_list:  0-31
/proc/2107/task/2196/status:Cpus_allowed_list:  0-31
/proc/2107/task/2197/status:Cpus_allowed_list:  0-31
/proc/2107/task/2198/status:Cpus_allowed_list:  0-31
/proc/2107/task/2199/status:Cpus_allowed_list:  0-31
/proc/2107/task/2200/status:Cpus_allowed_list:  0-31
/proc/2107/task/2201/status:Cpus_allowed_list:  0-31
/proc/2107/task/2202/status:Cpus_allowed_list:  0-31
/proc/2107/task/2203/status:Cpus_allowed_list:  0-31

So the cpus_allowed has a mix of isolcpus and nonisolcpus.

While this patch fixes the problem, there is a risk of missing other places
like update_numa_stats also iterates and accounts for tasks stats running on
isolcpus.

I will send a patch with a slightly different approach.
Request you to review and verify the same.

--
Thanks and Regards
Srikar Dronamraju