Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31;
Message-ID: <66d91674-fc2d-638e-ff92-7d3515e3aa08@intel.com>
Date:   Thu, 28 Sep 2023 11:44:45 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.15.1
Subject: Re: [PATCH v5 6/8] x86/resctrl: Sub NUMA Cluster detection and enable
Content-Language: en-US
To:     Tony Luck <tony.luck@intel.com>
CC:     Fenghua Yu <fenghua.yu@intel.com>,
        Peter Newman <peternewman@google.com>,
        Jonathan Corbet <corbet@lwn.net>,
        Shuah Khan <skhan@linuxfoundation.org>, <x86@kernel.org>,
        Shaopeng Tan <tan.shaopeng@fujitsu.com>,
        James Morse <james.morse@arm.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Babu Moger <babu.moger@amd.com>,
        Randy Dunlap <rdunlap@infradead.org>,
        <linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
        <patches@lists.linux.dev>
References: <20230722190740.326190-1-tony.luck@intel.com>
 <20230829234426.64421-1-tony.luck@intel.com>
 <20230829234426.64421-7-tony.luck@intel.com>
 <20c243fd-5068-b8a0-31d3-369b739ff1ca@intel.com>
 <ZRXCHVhpxhD6Joo8@agluck-desk3>
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <ZRXCHVhpxhD6Joo8@agluck-desk3>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZjlWNzhmTjhSUGQ3Y1lTU09heWxUdFE5bVlkREJMMFlRVzhnNkJrQllZRWlE?=
 =?utf-8?B?cUx4UGNHZFZmR0c0MEFhVE5ES29BV3dlbS9DTnVEbU40MXM5bTNRVS90eTk0?=
 =?utf-8?B?ekVpTmRPSU92aVVhT0gxY2oraEJRbVg1OE8wZVUzbjBreURDS0FvSHlidUpJ?=
 =?utf-8?B?U2lXK1ZocGdrZ2Jyb0YxQ1VvOHVxcFZHUDNQZUlvSzJqYUFyMkxpUUhhbWJC?=
 =?utf-8?B?cThyYlhBaFdLZjFtdExHbTl1bTVmdFFhb1Jya25ubXRtcGordHBNbEY2cVFM?=
 =?utf-8?B?ZE9lSklIZThyMDYwS3ZTcWhMcGdUNWVmWWM3ZE5WWm5ZdWg4K3E4Rk81YU1v?=
 =?utf-8?B?NTVacmxjQ1k0TlZBUW1Bb3krUDJkZzlvUk1xQlpzUGFmVWdGRkR4a1Flanlk?=
 =?utf-8?B?a3JtZHE3UUdYSmsxNmtDNVRBa0JtdS9CVzlWYU81NEFZRWQ3NUtpbXdTbm84?=
 =?utf-8?B?TVVFMmlwRW1GVjIzYW1welZBL0hYaDRZcEVNOVI5ckFHdWo3ZXhGaUg2cldS?=
 =?utf-8?B?c1ZSdXpaNXNLcStJSTkvakFhSGFvOVkwSW9nSEl5dHFZK0taaU5hTjEvQ292?=
 =?utf-8?B?L3JVWkNyd2RmcWc5RzI0aW9XZHRHRjlXTjVmK2piTUg3SHFaa3NaTzJQaURL?=
 =?utf-8?B?QWswOUZwR0lMVTN4RUpheGl1NWZmU0JXUXpMSG5wcW02N2c0S3doaGlkT05a?=
 =?utf-8?B?TnZkR1FESmpCcnQyZVp4M29OYmxhOHpmdWpnbHBTQ2IzWW04TnE4c2dyeFNU?=
 =?utf-8?B?d0dzUUVyUm8wQnRvNmxGWEUwV2dwZjEybE1wcndoSWxabHc0dkRWN1Y0bTJi?=
 =?utf-8?B?eW8xdjZhV2V5T0ROU1ZvUjNoTE5Tam1BbVNrVzlaWEFLWTduRDdrNmhpU0VP?=
 =?utf-8?B?RXJrQlJDRGJuN3E5OGM0ZEluU3pFRzJZUmxWb05EcHBxcWVqN3l6blM0MGZT?=
 =?utf-8?B?VXVqOHJab3puUUpJV0xzVVd1N3ZqV1FaaitUUC90d3piUG5uSldZaHVnQWJ5?=
 =?utf-8?B?TERrdHMxS1ZkUnZCMG9FZ3hhYXFjTXh4dTJVcC9Ba2pJRExuY0xPVUlKVTk2?=
 =?utf-8?B?S2cxNENrV2oraHpqUDlMREpTcERpR3RzZzNjbFZmWXB1RTRnSisxUkk4UHVW?=
 =?utf-8?B?UzFiRjlvczduaVIyNVE2VUZTUFI4VFhyUVBNZVpwVDF4cU5YZWlLTy9CY2Vt?=
 =?utf-8?B?NnhSUktCcWlLN2lqNERNL2xyWitQMkhKc21HNWxvZHRjUDJuSHB1ZThYNTdP?=
 =?utf-8?B?aVdleUhFeGI4VlhKME5IeXhBQm9kY0Fidk0zaWtkUzlJK2xvWG9MU3pka2t2?=
 =?utf-8?B?VnFOMHV3OTB0Q251eWFIN2dSMmxxdm9wTGc1SVo3TEYwcld5OUlUMy9wVTNs?=
 =?utf-8?B?aEF2bEpqUEh0NHRiUklPY1JRQUpZR0M4MnB0UG1nenNJMkF4Mmx5Z0tqN1Ay?=
 =?utf-8?B?MG5pR2g0dDlDbURzVEVYdlJ3enJyN2IyaGxMNE9QTGhROFFIayttcGQvL0pj?=
 =?utf-8?B?a3RSb2cwcGhNOTJBaHFCYStuanhNbW1qQUhWWjRac0pvTlJha0hxR3RBZ2Rt?=
 =?utf-8?B?NmcrQ1BhR2NzZitxZHdkbFNDaGRocmRRdWlkRjMxaklkVjdGQXltNEphR2hI?=
 =?utf-8?B?WXI3OUNPajQ3Q2hiOVcvdTlaMThPSVAwby9LdXpOZUVnWVFybEIvc1hHOWdM?=
 =?utf-8?B?d3BBVFcrYTlJalQwbUNPemJSZWFBY2xLckRCTWtBcTlnUzlhK1lHYWlJb0Ry?=
 =?utf-8?B?SFQ2SUNXTGpjREZLcnF2bWdrWjdOWTQ2K2ROajdmWnZldVZJdmxxcTdqTEhO?=
 =?utf-8?B?dWpGYmRCTGZGY05CaXpNRzBuMW81cHZORVRtbHhVQWFVU2MySG9NOUh0Mm5l?=
 =?utf-8?B?cVA1UWs3TnI5bmc4aENhUHJ3QUt6MldpN1hQMS9xZ3BEUHU3RTNHVEFkdTli?=
 =?utf-8?B?TnRBVUtnRUZNZWg4dGIxSkVLcUg3RkFaK1JKeHR0NS9GUXhsM2hDbWFxYjhn?=
 =?utf-8?B?dnpOYVJINjVjTFYwOUt5YXRObFR5Qy9JTXFFM2RqNkF5L1FHVGl0MEloUFJC?=
 =?utf-8?B?UVlCeVNOUHkxYVRSOE12b1kxNW9nUnV3bFdqL2Q3SmZMTE9Md1NoRXRIWWdU?=
 =?utf-8?B?VEVRb0RmbTg1S1VqbytTN3Ryczd1L1Fvc3d5cDVCVFE3OTBlZ0xWaHRJWjFW?=
 =?utf-8?B?VkE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ef037727-49db-417e-87a3-08dbc052fd46
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Sep 2023 18:44:47.3513
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: GEb7PAYTfaQodOonBv9iAYoi1mDFhIhvK5g+mmK4bOEpUYsj4aTqQGVeC+rmiLtEiVY/JdJSId6sjoXtnabW/LmxPybeQ/c1SSYqFxI1t6s=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB6816
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.3 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Thu, 28 Sep 2023 11:52:40 -0700 (PDT)

Hi Tony,

On 9/28/2023 11:12 AM, Tony Luck wrote:
> On Mon, Sep 25, 2023 at 04:29:44PM -0700, Reinette Chatre wrote:
>> On 8/29/2023 4:44 PM, Tony Luck wrote:

>>> +	if (cpumask_first(topology_core_cpumask(cpu)) != cpu)
>>> +		return;
>>> +
>>
>> This is an area I am not familiar with. The above code seems
>> to assume that CPUs are onlined in a particular numerical
>> order. For example, if I understand correctly, if CPUs
>> are onlined from higher number to lower number then
>> the above code may end up running on every CPU online.
> 
> This sent me on a voyage of exploration into early Linux
> bringup. There's a CONFIG_HOTPLUG_PARALLEL option to bring
> CPUs up in parallel. I have it set on my kernel, but I still
> see the "announce_cpu()" console messages show up in monotonic
> increasing order:
> 
> [    3.148423] smpboot: x86: Booting SMP configuration:
> [    3.148940] .... node  #0, CPUs:          #1   #2   #3   #4   #5   #6 #7 and so on
> 
> But, without solving this mystery, I realized this doesn't matter.
> Whatever order the CPUs come online was all completed long before
> resctrl is initialized:
> 
> 	late_initcall(resctrl_late_init);
> 
> So the order that resctrl sees CPUs is dependent on the callbacks
> from the registration with cpuhp_setup_state(). That happens with:
> 
>         /*
>          * Try to call the startup callback for each present cpu
>          * depending on the hotplug state of the cpu.
>          */
>         for_each_present_cpu(cpu) {
> 
> which is going to call in increasing numerical order as the bitmap
> of present CPUs is traversed.
> 
> If someone changed this, the only ill effect on the code I'm
> adding would be to set the MSR multiple times (which is
> inefficient, but won't break anything).
> 
> 

Thank you very much for investigating this.


Reinette