Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   "Huang, Ying" <ying.huang@intel.com>
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     Srikar Dronamraju <srikar@linux.vnet.ibm.com>,
        Mel Gorman <mgorman@suse.de>, linux-kernel@vger.kernel.org,
        Ingo Molnar <mingo@redhat.com>, Rik van Riel <riel@surriel.com>
Subject: Re: [RFC PATCH 1/2] NUMA balancing: fix NUMA topology type for
 memory tiering system
References: <20220128023842.1946583-1-ying.huang@intel.com>
        <20220128052345.GA618915@linux.vnet.ibm.com>
        <87czkctiz9.fsf@yhuang6-desk2.ccr.corp.intel.com>
        <YfQIMmbY7nHusQRK@hirez.programming.kicks-ass.net>
Date:   Sun, 30 Jan 2022 09:07:17 +0800
In-Reply-To: <YfQIMmbY7nHusQRK@hirez.programming.kicks-ass.net> (Peter
        Zijlstra's message of "Fri, 28 Jan 2022 16:13:54 +0100")
Message-ID: <87sft6rpyy.fsf@yhuang6-desk2.ccr.corp.intel.com>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=ascii
Precedence: bulk

Peter Zijlstra <peterz@infradead.org> writes:

> On Fri, Jan 28, 2022 at 03:30:50PM +0800, Huang, Ying wrote:
>> Srikar Dronamraju <srikar@linux.vnet.ibm.com> writes:
>> 
>> > * Huang Ying <ying.huang@intel.com> [2022-01-28 10:38:41]:
>> >
>> >> 
>> >> One possible fix is to ignore CPU-less nodes when detecting NUMA
>> >> topology type in init_numa_topology_type().  That works well for the
>> >> example system.  Is it good in general for any system with CPU-less
>> >> nodes?
>> >> 
>> >
>> > A CPUless node at the time online doesn't necessarily mean a CPUless node
>> > for the entire boot. For example: On PowerVM Lpars, aka powerpc systems,
>> > some of the nodes may start as CPUless nodes and then CPUS may get
>> > populated/hotplugged on them.
>> 
>> Got it!
>> 
>> > Hence I am not sure if adding a check for CPUless nodes at node online may
>> > work for such systems.
>> 
>> How about something as below?
>
> I'm thinking that might not be enough in that scenario; if we're going
> to consistently skip CPU-less nodes (as I really think we should) then
> __sched_domains_numa_masks_set() is not sufficient for the hotplug case
> since sched_domains_numa_levels and sched_max_numa_distance can also
> change.
>
> This means we need to re-do more of sched_init_numa() and possibly
> re-alloc some of those arrays etc..
>
> Same for offline ofc.

Got it!  It doesn't make sense to create schedule domains for CPU-less
nodes.  I can work on this after Chinese New Year holiday week (the
whole next week).  But if anyone want to work on this, feel free to do
that.

Best Regards,
Huang, Ying