Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 4 Nov 2022 15:21:29 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     Dave Hansen <dave.hansen@intel.com>,
        Zhang Rui <rui.zhang@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        "H . Peter Anvin" <hpa@zytor.com>,
        "Peter Zijlstra" <peterz@infradead.org>, <x86@kernel.org>
CC:     <linux-kernel@vger.kernel.org>, <tim.c.chen@intel.com>,
        Xiongfeng Wang <wangxiongfeng2@huawei.com>,
        <liaoyu15@huawei.com>, <oliver.sang@intel.com>,
        <ying.huang@intel.com>
Subject: Re: [PATCH v1 1/2] x86/tsc: use logical_package as a better
 estimation of socket numbers
Message-ID: <Y2S9eTeOch+TYmks@feng-clx>
References: <20221021062131.1826810-1-feng.tang@intel.com>
 <f27e4b3f858890c657df9a7d6f34dc2d60b89757.camel@intel.com>
 <63dca468-c94d-844a-5b19-09c03cf84911@intel.com>
 <Y1ZArguS/rVEjXOb@feng-clx>
 <397f513f-9273-76d1-a0ba-9d1d403020c5@intel.com>
 <Y1eW2L5y7jx69v0g@feng-clx>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <Y1eW2L5y7jx69v0g@feng-clx>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?yS3OKCsAD4eZnMMKnwBbY3o60n8XFEkh0Kg5oimuX+vEnq2wDrXQkxWnZUly?=
 =?us-ascii?Q?hggKe5lsXGha8aU7R/D/oZXFv4B0uTG13zmzkBUbQgEBPI3+jmtql/FNwL/m?=
 =?us-ascii?Q?KYLNlWdf8kXhLx4HI0mCMEwhFIhJmI6QkHy5xT7hP7CB22HtS3aUd8TvtabF?=
 =?us-ascii?Q?5+WY+TvuUKt6QCQ4BW6z6lMExqGuSGBow1h6+qRqyCEDnCctepoNp4CgEGW4?=
 =?us-ascii?Q?9brj7tZuOWRQ3ZVDhLEJeDmDfucjMn/9dfZmTQfTHLTknxzk8h4YbdIk8rOb?=
 =?us-ascii?Q?gLmi6eS/1GG8+wmrB7GScSs5us8szWQ/+LpRs51oWITUR7L5fqH9M00lSgHE?=
 =?us-ascii?Q?CyB9Mh7aT8Pvx7QbEJkQDmCoJ0stfmct6FfSyEHSeIy8KuZ932/rR262znsf?=
 =?us-ascii?Q?/KbRFq/ijZS5EwxBIKT7nN8J06O8T5tjZQg6fEO1SxZbkgwNvlQoeOTu/oLu?=
 =?us-ascii?Q?qqIWhnggPN+DR3vNswJA2oG14USDnfskwth1+OeXf37h3ATCUvU//KTJ0W6Z?=
 =?us-ascii?Q?rnM5joiNWqNg+pfEiUc6K/LnYQUMC13Z/NSNKVvV9aqWVnKQxDtZhfP55NPI?=
 =?us-ascii?Q?lcIGCnGuYz04tO1Iltu934rN3HgKleZErOsm6EJAqDaW138FVlAeyoqEh3jF?=
 =?us-ascii?Q?K+hDdF8RLysQ3xYRBdYRtLf4aOr3NKcdfIMg0JV/egjEiKf+DHIAev5OKuCR?=
 =?us-ascii?Q?Jy+Ndpl5G43F7rKKVrcPsPtrEXSF+FhDQU0kwHsMmvZfqXaXy4qim+BV3UGp?=
 =?us-ascii?Q?Bf0rEJt1BQVwWnIo3I8mrrSG/sKvv1GIH4vSSIJHdYPgyD1IiISuR1c4vB7B?=
 =?us-ascii?Q?r7NQH11af/KYuFv6xAYTbWjs/j2kN/66Ksn/BTUzSD6PsVsazyewWDskcgUt?=
 =?us-ascii?Q?Fy7wM4+TEAnZil0eb0NOquE7TEgJFIPCYVEkzEvOVF+uJLtqMFf2zrEawtHT?=
 =?us-ascii?Q?RtzQf/9Pf5LuMfKRf5SwLQhuy31mUvpf3+bmOiJ58eVJbTUcr0bClUEn59yC?=
 =?us-ascii?Q?5yT3F4F+y5MdBtg+K/DcqodDKIWqUcmzW5a1Cvj4C6Amh6YbCqODal9gj0d2?=
 =?us-ascii?Q?frT55PWGb/5CqIfld1aeupuXaY27AJtwr/Lqur8L+r2ig2dRtrooPEEq8cIM?=
 =?us-ascii?Q?HHIYv8mChpzUZw4ZVCX9Ww/upjzUuLaAx8ONpl+P6kMSrTcQpjn7hyrWptzS?=
 =?us-ascii?Q?uUDntDuYEQ+ipId0kwwLPQz0DjcKQugLO1dSmD13TvBS5flGQupJBeNZsdVT?=
 =?us-ascii?Q?BaUmynwgXbVFFV+0HWcVF6s2a/wuzpJCZxHX8NHUXKGctsz7kdjX/npygETa?=
 =?us-ascii?Q?NY7bmXl0xQUQvAqYaeNzNN0LDavX/EmJMTIZnP/plp+di0HBaL544eNiz2ym?=
 =?us-ascii?Q?x1oUV9+Np1PXdmYT/IMWd4JjlKgLUmAbQjCmUNTiYfnMOXCLhk8h9vH8NpNe?=
 =?us-ascii?Q?QUEPBRxwLDyG1ip1qL8WBhxAEE+vEkQExJWWp2yXBElgwlra5pcRu0V7VMlm?=
 =?us-ascii?Q?cKIZFmq963W63cTZT3WYZWz4KawePJDjNHO03ZskmWODclEGqdCY/ffH11C1?=
 =?us-ascii?Q?Qx4HzFcif7tkhYcKjDcwYTDWkH6pWc6VWhTgcKxB?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 5639dbda-32e3-4bf1-cca8-08dabe35a391
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Nov 2022 07:24:41.5475
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 8jgLJ7HGEN7rrZxXQfbIpg5dCmQCGzLyUpTuaCPZASrFN7xqWZzrVMeVVkbBWuMXRoZBwr1ElQtsLIFd07rFHQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR11MB7536
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-8.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Oct 25, 2022 at 03:57:12PM +0800, Feng Tang wrote:
> On Mon, Oct 24, 2022 at 08:43:33AM -0700, Dave Hansen wrote:
> > That's too wishy-washy.  Also, I *KNOW* Intel has built systems with
> > wonky, opaque numbers of "sockets".  Cascade Lake was a single physical
> > "socket", but in all other respects (including enumeration to software)
> > it acted like two logical sockets.
> > 
> > So, what was the "real" socket number for Cascade Lake?  If you looked
> > in a chassis, you'd see one socket.  But, there were two dies in that
> > socket talking to each other over UPI, so it had a system topology which
> > was indistinguishable from a 2-socket system.
>  
> Good to know and thanks for the info.
> 
> I have to admit I haven't checked a server's internals before, and
> thanks to Oliver for helping checking some Cascade Lake boxes of 0Day.
> 
> In one box where 'lscpu' shows 4 sockets (96 cores in total), it does
> only have 2 physical processors in the chassis, just like you
> mentioned, it has 2 dies for each processor. And in another box,
> 'lscpu' shows 2 sockets (44 cores in total), it also has 2 physical
> processors but with much smaller size.
> 
> And fortunately the 'logical_packages' for these 2 boxes are both
> the correct value: 2.
> 
> > Let's just state the facts:
> > 
> > 	pr_info("Disabling TSC watchdog on %d-package system.", ...)
> > 
> > Then, we can have a flag elsewhere to say how reliable that number is.
> > A taint flag or CPU bug is probably going to far, but something like this:
> > 
> > bool logical_package_count_unreliable = false;
> > 
> > void mark_bad_package_count(char *reason)
> > {
> > 	if (logical_package_count_unreliable)
> > 		return true;
> > 
> > 	pr_warn("processor package count is unreliable");
> > }
> > 
> > Might be OK.  Then you can call mark_bad_package_count() from multiple
> > sites, like the maxcpus= code.
> 
> This should work! we can just add one more check:
> 
> 	boot_cpu_has(X86_FEATURE_TSC_ADJUST) &&
> 	!logical_package_count_unreliable &&
> 	logical_packages <= 2 
> 
> And when some new case leading to a imprecise 'logical_packages' is
> found in future, we could just apply to this to it.

Hi Thomas, Peter and reviewers,

For estimating socket numbers, there are quite some BIOS/kernel
settings that can affect its accuracy, like:

* numa emulation (numa=fake=4 etc.)
* numa=off
* platforms with CPU+DRAM nodes, CPU-less HBM nodes, CPU-less
  persistent memory nodes.
* SNC (sub-numa cluster) mode is enabled
* 'maxcpus=' cmdline limiting onlined CPU numbers

Ideally, BIOS could provide that info in some MSR or through
CPUID, as it knows most of the information before transfering
to OS, but that's just my wishful thinking for now. 

And we checked several ways for estimating socket number:
* nr_online_nodes
* SRAT table init(parsing node with CPUs)
* 'logical_packages'

And 'logical_packages' is a better option after comparison,
as it works for mostof the cases above except 'maxcpus='
one, where the 'logical_package' could be smaller than the
real number. Dave suggested to explicitly skip the check
and warn.

We plan to use 'logical_packages' to replace current
'nr_online_nodes' for estimation of socket number, any thoughts
on this, or some suggestions? thanks!

- Feng