Received-SPF: pass (google.com: domain of linux-kernel+bounces-37412-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Wed, 24 Jan 2024 09:19:00 -0800 (PST)
From: "Lameter, Christopher" <cl@os.amperecomputing.com>
To: Huang Shijie <shijie@os.amperecomputing.com>
cc: gregkh@linuxfoundation.org, patches@amperecomputing.com, rafael@kernel.org, 
    paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, 
    yury.norov@gmail.com, kuba@kernel.org, vschneid@redhat.com, 
    mingo@kernel.org, akpm@linux-foundation.org, vbabka@suse.cz, 
    rppt@kernel.org, tglx@linutronix.de, jpoimboe@kernel.org, 
    ndesaulniers@google.com, mikelley@microsoft.com, mhiramat@kernel.org, 
    arnd@arndb.de, linux-kernel@vger.kernel.org, 
    linux-riscv@lists.infradead.org, linux-arm-kernel@lists.infradead.org, 
    catalin.marinas@arm.com, will@kernel.org, mark.rutland@arm.com, 
    mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, chenhuacai@kernel.org, 
    jiaxun.yang@flygoat.com, linux-mips@vger.kernel.org
Subject: Re: [PATCH v2] NUMA: Early use of cpu_to_node() returns 0 instead
 of the correct node id
In-Reply-To: <20240123045843.75969-1-shijie@os.amperecomputing.com>
Message-ID: <4a13353c-cf4b-a388-5776-389c61c63ec0@os.amperecomputing.com>
References: <20240123045843.75969-1-shijie@os.amperecomputing.com>
Content-Type: text/plain; format=flowed; charset=US-ASCII
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?H6NdgnWRJOfUbn1r6TVJsmVhRuXealrDl7I8c7BgggiMOcNCgAf+3cfZ6kmO?=
 =?us-ascii?Q?KsXvRJjXkGVmjF4v51KMvhOg+cxdCPq4Q81zDB8g7wQVsNyS5ORNwJp1KQqw?=
 =?us-ascii?Q?N1Z0idggreBnOHRJJ8WypdV0LOtV7diqTmt4JYNtGY1TJ1//PZv0PstNr0gr?=
 =?us-ascii?Q?esOwSF22xS5sFCS0b5PEPA8adrIrUtUNiNyYT7oqrj4kqvJUZTlD5By9kOEn?=
 =?us-ascii?Q?q6zUtdfumekcrqObpwiYg4gCsvNU9zw0iPabUQqfON17xKC53Smj96y013Cy?=
 =?us-ascii?Q?xyeAzLs2GUNTZ6g/24A220/lc0CPJUSbOn5Rxj7LbJF8XQ9Oui2osWUk5hYt?=
 =?us-ascii?Q?Cdhg1BQkiIWOBD1pCgBKuYUHhVDwornVa5c5DtSj7Z9L4fUyN5VbZJuo6+5n?=
 =?us-ascii?Q?FB1Ghw/Ll2T8RhubQ9QV9rvDRCM4VIKMXSDBurFGMVX9Ph92ep6a8TNI3ILL?=
 =?us-ascii?Q?L7fhYUejlcwrR/azcr9EeyJWel79pA7gKVqrFPKncfpg2/Va48pf/IVa7KSp?=
 =?us-ascii?Q?xAE8KpDFNOIRS/8YrC84xsHT0UcFLYjyfRR1XMRJYQcUXnFgAVue5qFSKdhD?=
 =?us-ascii?Q?LBlWPEGhwTxsatWRjyrvwFyi9/y0sqeL+GZPOOJbu0RHVYKc4P/uW1l/qmSo?=
 =?us-ascii?Q?IeKh5fUJCCJ25QtZAIumAKuBU2jz62CmzukBcLb27uss9EJVLvmnBrUB6IJq?=
 =?us-ascii?Q?pDSdSS16wTQSeIB5pZJ1KwgagYt1mIQ7fzLne94K9BcZcVE+vrenhFw74KzB?=
 =?us-ascii?Q?1V084oXgNxX0+qcYSh8sFFU897AsRGRt+zCPg8UY0MWDsBqzt5XctpuIjc/W?=
 =?us-ascii?Q?YtbhxZEdxfhvP8i7du6txlNFMVffanwbN4tvatFw92zJGDESnnJ4NeRLLm/B?=
 =?us-ascii?Q?PTGzN92Awl68bVvWu91RnGdujf7Br9/f3lTgtcxCiGDlUCEgy1cWi/v+nIla?=
 =?us-ascii?Q?+vDBVOR2NLwU6AZODJ4MiMDApm3+KNrCCfRAqAWPpWpLDW2i3Pphia3K+ewU?=
 =?us-ascii?Q?YNAVymJhSKZbtTatRUZLN0eIINooVh27tuCG1f6hKquN7aBGDWJPwIPMN2WL?=
 =?us-ascii?Q?3qcODodXOGr5vD2cknB2I9ue8CKS22KdC3mmZvzfAeP++Ov6c9qjoeJ5Mbyn?=
 =?us-ascii?Q?y7i7tnwbPx7cSC4CfOn9I37GL1VhqxIlpS4J/3l3iaUYQvECk+cPeHHFHFc1?=
 =?us-ascii?Q?qRQqDn0z7BbOVttPOYSgvQT0CGTkOY6lAujyhGbTALzGX5qbQ1d1ksBHbxU/?=
 =?us-ascii?Q?Mh/jXkX4/2Q1e4yaj+oT5ZIE8PUk9SMuBKO+tCawdqRmU+YHP4I6TGbXIYme?=
 =?us-ascii?Q?gnrE/sV6nMRtKAhSVbS7M8RvvMJj/E2U442cWITJm+A5ob6/+gGexcdE/zoK?=
 =?us-ascii?Q?ofnY5F/dG8oZzEqd2klmLagnPHF/jZLGlGQm9w2VMMQSu72eJImbMScDsKKt?=
 =?us-ascii?Q?9oLq8z1l3sg0N0NWef3XU7VLrPYma3yZK8Lt/whXPK9Ziju3a0GstyQncNW8?=
 =?us-ascii?Q?Ot4C0fDTMk/3ns+TO4HtLxMup0NLkEOQBDCh4xLGeCKXHb6LsHRqAMwpgZVA?=
 =?us-ascii?Q?U8gTML82mQfVzC15nXGW2wTW6MQFukkrN0KITKyvGww4jVYaNOZB7eLo2k+z?=
 =?us-ascii?Q?gg=3D=3D?=
X-OriginatorOrg: os.amperecomputing.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 451d01bd-5263-49e2-6a6c-08dc1d00910a
X-MS-Exchange-CrossTenant-AuthSource: DM6PR01MB5259.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2024 17:19:05.6274
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: +KMtxBTJGu0dCKRUx6l8us5K3cK1D4eGG3V0PeDWSUexZBgXDex9/Hd3CgRd4IQt/+00UUYMd7NZl/OXAKRyZkWlv2Vh8gaZqJ/khGiO2hE=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR01MB6479

On Tue, 23 Jan 2024, Huang Shijie wrote:

> During the kernel booting, the generic cpu_to_node() is called too early in
> arm64, powerpc and riscv when CONFIG_NUMA is enabled.
>
> For arm64/powerpc/riscv, there are at least four places in the common code
> where the generic cpu_to_node() is called before it is initialized:
> 	   1.) early_trace_init()         in kernel/trace/trace.c
> 	   2.) sched_init()               in kernel/sched/core.c
> 	   3.) init_sched_fair_class()    in kernel/sched/fair.c
> 	   4.) workqueue_init_early()     in kernel/workqueue.c
>
> In order to fix the bug, the patch changes generic cpu_to_node to
> function pointer, and export it for kernel modules.
> Introduce smp_prepare_boot_cpu_start() to wrap the original
> smp_prepare_boot_cpu(), and set cpu_to_node with early_cpu_to_node.
> Introduce smp_prepare_cpus_done() to wrap the original smp_prepare_cpus(),
> and set the cpu_to_node to formal _cpu_to_node().

Would  you please fix this cleanly without a function pointer?

What I think needs to be done is a patch series.

1. Instrument cpu_to_node so that some warning is issued if it is used too 
early. Preloading the array with NUMA_NO_NODE would allow us to do that.

2. Implement early_cpu_to_node on platforms that currently do not have it.

3. A series of patches that fix each place where cpu_to_node is used too 
early.