Received-SPF: pass (google.com: domain of linux-kernel+bounces-105625-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Mon, 18 Mar 2024 09:16:49 +0800
From: Feng Tang <feng.tang@intel.com>
To: Dave Hansen <dave.hansen@intel.com>
CC: Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>,
	Borislav Petkov <bp@alien8.de>, "H . Peter Anvin" <hpa@zytor.com>, "Peter
 Zijlstra" <peterz@infradead.org>, <x86@kernel.org>, <paulmck@kernel.org>,
	<rui.zhang@intel.com>, Waiman Long <longman@redhat.com>,
	<linux-kernel@vger.kernel.org>, Dave Hansen <dave.hansen@linux.intel.com>
Subject: Re: [PATCH] x86/tsc: Use topology_max_packages() to get package
 number
Message-ID: <ZfeWAQoYLFLnhbmG@feng-clx.sh.intel.com>
References: <20240315112606.2248284-1-feng.tang@intel.com>
 <ce02f1a8-870f-41bc-8650-4bd6103f9637@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <ce02f1a8-870f-41bc-8650-4bd6103f9637@intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?x79HhKRMMUQuA0oWwMBaBmlDijZIj0FpH/pzLmBCIS1ah1Da/v7ikYZlThY9?=
 =?us-ascii?Q?XiON2/gmsAyJKPgynZjfrbxIStds/dYJYVY65GgdPMJdHP0l4Q6g8n4yCfYQ?=
 =?us-ascii?Q?BvR/U/KaFtYj4FsXlZV4vkEyrzylAZnmiid5IB8PqbBZbVegGMXqEgOeTFQq?=
 =?us-ascii?Q?DXUS2jzeOJhKVHZey5Er9BEDyC2kgq1vTXWljMQ0HMSCB2J5iEEIe55ZhNcK?=
 =?us-ascii?Q?Ha+cC+9A8DKZncyyulkQDSKdbSmsGlaAgijQEuiCfnVsXvGDxXQ2uI1bBPS3?=
 =?us-ascii?Q?7CoRM3xr7nFr4PENSLM7woVoJ+S4juzSo6YUVWb97bFy3PR8HPkSQxu2WUYb?=
 =?us-ascii?Q?P7YyIxmf2D/9gTg6B04UFweeSf9MG3X8CawKH5p5fK//jChvaPtROBWvtfxW?=
 =?us-ascii?Q?HBNhmA0pYHm1DMN6arC2Vje0KpuQqrHlTLy2xkXn2i2oUK7u51/Bnq/Bco7g?=
 =?us-ascii?Q?B+w6iAt3jsO2kTO7xnRrwbaRpK0YuiI9lhimlI08LSL6+YPZRUrVo8QnhEHO?=
 =?us-ascii?Q?rlNF5kKUqOJVIsWDFrJFXtJQHzysRPVwc+8DGuHGo0dAFKpR6kV9U1+Qwyo5?=
 =?us-ascii?Q?Gkooss/SGAU95cumDAWpjN4f2ZF7AE1Z08Q8iEVGeJ00fiLsf4Z0/+0+5yK3?=
 =?us-ascii?Q?4D15AfedSnqnhi51Mt7nZi3NrlTVbS8df1KAXRNuDXhfe13Q6NIfODsuYGpJ?=
 =?us-ascii?Q?fRZK8b19AGk2+V4PXeQzk4SRs/35hgsNMo/NTye6d7QuizNC72gxb+Gs6jzu?=
 =?us-ascii?Q?qekffBrxlLwivutz97fdJ1DYH+65544TIVld2Auw68C6Z2cY9VSWbFWJ7fMF?=
 =?us-ascii?Q?W7UwgEmABMLkNuk4dnUjQA7oAw/AOmFIfkyyXrr7BtAkc2LOGPc7FOXItTpu?=
 =?us-ascii?Q?/Qdz9Rr8FO1HG9ff/9fZKkSpvPQxe0cMzNk6DnZ+r/KYwMcqRHallKlcz/7a?=
 =?us-ascii?Q?LaYcW+ozcMCIyEENF80VHjAc+YIraRaIRemU2Pk8qWGuD/J6Nsq7cZkHM9hp?=
 =?us-ascii?Q?UPSNhkO+uFoqxo29VF/XR2ShZmxM6NrSDWus2+x+jOo+8k/cDSczSFfmYZUB?=
 =?us-ascii?Q?1ECSrAJIDZU6u+7UoQ/DfTKQVuvGeWX4fgbnYGwI+ZE/CLKTEEjuJvp6X8a9?=
 =?us-ascii?Q?rSpQGQlk+eQJEyMCyHqWaDiSZ1bTpvD8dwsrDWw3q/OwWv+dpxlTrnuo7T/7?=
 =?us-ascii?Q?wyzjkuB5Y9IFSadT19zfhPAJ3gskz+S0AQscwo43+USo4eV8NJGh+QeOSTC5?=
 =?us-ascii?Q?sJ3QEDmQG8LirslBkHYqFq9rwAimg5Tm85Gp64V7/TVuOnI/Y7BcBY7Rw3zO?=
 =?us-ascii?Q?QbSKSfah6Kwfb5AGnjv8XgYsq2eKl/BUL2lOkExonhb0y3UKZf0skDV4atZt?=
 =?us-ascii?Q?V4HKerPIHDPChcOPr2AbiPji5V5N1I/zT0rk6EOXAZR9rsRnJjddZlQ7AOrz?=
 =?us-ascii?Q?iDzOFHvEYPxQ8nSL4HigdiNJs1S+Xdxt3Q97g7B7qKUdlnqvVsw9TUvgl3Ed?=
 =?us-ascii?Q?JNaLW40mk/ZcqAo+1IzYjkufpGRI5xwobneRhhPgHpLCP3/9QJhSj0p5n7vr?=
 =?us-ascii?Q?xTmhsbKTSqUkrzUp75LxYouISFcMOk0y4RVEIStG?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 4e6c2e13-26a2-458b-815f-08dc46eb09e6
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Mar 2024 01:30:47.8038
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: nM5Ru/wx9evIFeCl4sCfl8mqgZugfd3uHmVuADrq7y+7CQ2ZlAwiu2BfAK/Z90hIPbAjr5wSPDHytyuUbyTxhA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB7552
X-OriginatorOrg: intel.com

On Fri, Mar 15, 2024 at 10:58:38AM -0700, Dave Hansen wrote:
> On 3/15/24 04:26, Feng Tang wrote:
> > Thomas' recent patchset of refactoring x86 topology code introduces
> > topology_max_package(), which works well in most of the above cases.
> > The only exceptions are 'nr_cpus=' and 'possible_cpus=' setup, which
> > sets up the 'nr_cpu_ids' and rejects the rest of the CPUs, and may
> > cause topology_max_package() less than the real package number, but
> > it's fine as it is rarely used debug option, and logical package
> > number really matters in this check. So use the more accurate
> > topology_max_package() to replace nr_online_nodes().
> 
> In the end, we have a bunch of hardware enumeration and then a bunch of
> processing on top of it taking CPU hotplug support and kernel command
> lines into account.
> 
> The hardware enumeration is relatively simple.  The processing the
> kernel does on top of it is complicated.

Yes!

> 
> > diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c
> > index 5a69a49acc96..87e7c0e89db1 100644
> > --- a/arch/x86/kernel/tsc.c
> > +++ b/arch/x86/kernel/tsc.c
> > @@ -1252,15 +1252,12 @@ static void __init check_system_tsc_reliable(void)
> >  	 *  - TSC which does not stop in C-States
> >  	 *  - the TSC_ADJUST register which allows to detect even minimal
> >  	 *    modifications
> > -	 *  - not more than two sockets. As the number of sockets cannot be
> > -	 *    evaluated at the early boot stage where this has to be
> > -	 *    invoked, check the number of online memory nodes as a
> > -	 *    fallback solution which is an reasonable estimate.
> > +	 *  - not more than four sockets.
> >  	 */
> >  	if (boot_cpu_has(X86_FEATURE_CONSTANT_TSC) &&
> >  	    boot_cpu_has(X86_FEATURE_NONSTOP_TSC) &&
> >  	    boot_cpu_has(X86_FEATURE_TSC_ADJUST) &&
> > -	    nr_online_nodes <= 4)
> > +	    topology_max_packages() <= 4)
> >  		tsc_disable_clocksource_watchdog();
> >  }
> 
> I know there's some history here, but the changelog itself is not clear
> about what the problem is or how the patch solves it.

OK, will improve the changelog. The problem is nr_online_nodes() is
not a good option for get package number, it is mostly a memory node
concept, and easy be cheated by different kernel cmdline setup like
NUMA emulation and hotplug tricks, while it had an advantage of being
availab early before TSC get initialized. Thomas' patchset improves
the topology code, and provide a much better topology_max_packages().

> 
> I also kinda dislike the comment talking about "sockets" and the code
> talking about "packages".

Will unifiy to use 'package' term.

> I also did a big *gulp* when I saw this:
> 
> 	#define topology_max_packages() (__max_logical_packages)
> 
> and:

Latest code has dropped this.

Thanks,
Feng

> >         /*
> >          * Today neither Intel nor AMD support heterogeneous systems so
> >          * extrapolate the boot cpu's data to all packages.
> >          */
> >         ncpus = cpu_data(0).booted_cores * topology_max_smt_threads();
> >         __max_logical_packages = DIV_ROUND_UP(total_cpus, ncpus);
> 
> Because Intel obviously has heterogeneous systems today.
> 
> So I'll buy that removing 'nr_online_nodes' takes NUMA out of the
> picture (which is good), but I want to hear more about why
> topology_max_packages() and '4' are the right things to be checking.
> 
> I suspect the real reason '4' was picked was to give the calculation
> some wiggle room because it's not actually all that precise.
> 
>