Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [tip: sched/core] sched: Add cluster scheduler level for x86
To:     linux-kernel@vger.kernel.org, linux-tip-commits@vger.kernel.org
Cc:     Tim Chen <tim.c.chen@linux.intel.com>,
        Barry Song <song.bao.hua@hisilicon.com>,
        "Peter Zijlstra (Intel)" <peterz@infradead.org>, x86@kernel.org
References: <20210924085104.44806-4-21cnbao@gmail.com>
 <163429109791.25758.3107620034958821511.tip-bot2@tip-bot2>
From:   Tom Lendacky <thomas.lendacky@amd.com>
Message-ID: <9e7b0c92-5a3b-8099-8c69-83a9d62aced4@amd.com>
Date:   Wed, 20 Oct 2021 08:12:51 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.13.0
In-Reply-To: <163429109791.25758.3107620034958821511.tip-bot2@tip-bot2>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?U3l5NjB4Q29jbXZIVlE0UjVkYkhnaDdDWWxnNnhlcHpoUjIxTFpzemowRGNo?=
 =?utf-8?B?UTFueW13cGoxci80anpRV0MzRG5oZW9OWDJJa1Q0NjRNbkZOZ0lSaCszYnAz?=
 =?utf-8?B?UzY5amRMclZNN20yOTVGaWtudE8zbXdNZWphalpwNjJ0ODZBblZRSGdDZDAv?=
 =?utf-8?B?NEFON3FqRVpuWWJnS0hqaVdLdXN4SFI3QXZKZ3JRcW94R1ZicUZjNVBubDBN?=
 =?utf-8?B?bVZ5ZTVzN0NmcFViMHRKdlZBdmNnQ2dWMWNUcDdPY2FPdmk4Y1FHNkhjVHAw?=
 =?utf-8?B?NFczY2VKOEMrTnhQWUlqWUdXMGt4emhrbkZEQkVFa2FxcDlqUFpCMnQyaWhw?=
 =?utf-8?B?aTZrSzB0bWZqb3pveFdPSElWeHBybEhtcWZvVU40RkhhVGx1dmJlUUdUbEFF?=
 =?utf-8?B?SkhGSTliR0hjS2huYTN2a24rOE05aDZaL1ZZTkRPQkNrdkFWWGVRcFhKakZi?=
 =?utf-8?B?WFVvMDV5M3ROL3hEZDJZcTFBS2tlYVBRcW1peVlqMGRybURUZkZCVGZBWFBu?=
 =?utf-8?B?SWtsb2xJVDNWWU9rL0s3RHNySC9iK2dtVTF6czl2cVdOdzhWUDZsUlZUMkVt?=
 =?utf-8?B?Smk2cXY0KzBsZVdMalZGSXhhU3U2TTRXUTd4c0J2UytQN2U1MzdhdjFDNDFj?=
 =?utf-8?B?QW1zaUpYWmdxbkRtcUUvaFVndmdFaGs0Z0NwTUlUUDEySy94VXljWll2ZmQv?=
 =?utf-8?B?ajNnYUtyRUVaK0d5VkdRYXFiMEIvL25VcnoxRzNKZUFtOThZSHhpQnhBZXI3?=
 =?utf-8?B?RFAyWWRjaVo3NW1jNkFvL3JBYlVxU2V4TUhHQWQzNGJLYkZJcFRVYlp4bFdk?=
 =?utf-8?B?NTNUVkJjZG5oeENud1BTVmJWblEyRmJXQzQydVRRcXROazJHNUFFL3ZGZEJD?=
 =?utf-8?B?Y0tWM0FMLzNJei9aa1lacFYzWExLUjFUVkd3QzVKWWIrQStLeHdDVjRUbDh1?=
 =?utf-8?B?UWZtenVDR3N5WFRvcWlhT0xESUdGQlBXSHcvZlhRc3JQK25sN3RtaUlGMzhM?=
 =?utf-8?B?MWt5a2VSRkNBb2tiVk1sZ3A5S2xVd0ZxQWtJYTNLYU45eS85QmRtMUx4VER3?=
 =?utf-8?B?bTgzMUdkTmF5ekxZaFFWekpmY3paalVnOGY2a1RvNWw3dzQ2VkJEa1J5VWNw?=
 =?utf-8?B?cWNhRWlmSmJlVldOMXVxL3I4bnUwK3FBYkNTZXhLZmY2cWVIUmVwWTg3SEE1?=
 =?utf-8?B?NUxNMjAzVWppWWpCeWF0U0RmRjlhY1hIc3NFV2tKYTY5SjQwZldneXpoZmVs?=
 =?utf-8?B?bzlRSk1YK3RQb2dVeGY0U3I4N1hENzE0elJzV1NDZEpIWDBJRjVTUlJOK0t0?=
 =?utf-8?B?V2JkaVBWUTJQK0VwaTVDQnB0VmxhQjlNbWZnbHYwdGNETkRJWURtZEtTSk05?=
 =?utf-8?B?UUFqMk5IaGhSc2p0UytuUTNrZmJDUWhFaVhUcVJHanplMlgrV2VXTWltYUFu?=
 =?utf-8?B?VHZEODZ5U2d2SUhrL3M1TVFWZVNQbFo5Vy8ycGplR2Nzamp5NGJxTEVNMy84?=
 =?utf-8?B?U0JTb1YycTRYOFF5aStZTE45TWtSNzltSVNEZGhPRDlmU0J1ak9CWENueDcy?=
 =?utf-8?B?OHNJbnhTNzFkU05KL21UVXFNY2lqWW95b3o4U3NlREZQMjN4bWc2aFJINVBs?=
 =?utf-8?B?ZG1oei8zQzJqeWFTdmJOT0wxL216bW5YTUtMVnVobVUzWUNtVWlxQWxwZEk0?=
 =?utf-8?B?Y3NGeHZDRisxNVZ0azZXZ0ZmYXlhZUxoWlFYVW9Sb094dVJzQm5zSEJlT2Fs?=
 =?utf-8?B?aitySmZsY2kvVmcxUHVtUHNsQ1IwRzR5SHhsY1VVZ0wrM0dRb1p5ckxkb24y?=
 =?utf-8?B?NnhtU2xCdXA3UkRqTFNPVGludDdYcFhvdWp4S2luUlBOYmF1V1hadUwwOUhB?=
 =?utf-8?B?WUVnNTk0eUZrc3k0YU5jaWlIY0o1MVlvOEFJbTNLQTRIN3Y5NDA1MjhXTVNE?=
 =?utf-8?B?SExNNndQOG0yMS8xSnh4czg0WUpYTm95cE5uUmdGaEJ2elRoTTBpdFF2VmFq?=
 =?utf-8?B?SVdZejZ4TzE2eElKV2lsVmpibmhHb0ZiNkpMV3NEQVR0a1BIZXhoMDgwTUhC?=
 =?utf-8?B?Uk1SaDhrV0p4VCtRV1FmZEJpT1NyN1RxQVg5N1VrNzJiWTlVOElCejVjeitO?=
 =?utf-8?B?K1oycDU2YkVGRmprYWV6QjhyM1dQbUY5Z05GWThMS2ZnRVZmYWw5YUlKRHdV?=
 =?utf-8?Q?NJdDQ2CJNUV/jGg/6Tl6aSs=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9bd17d29-43cd-495e-fea5-08d993cb5355
X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5229.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Oct 2021 13:12:53.8152
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 0Th7q4xTV1if94NmBjeXWdI/jck+zQSiPGakiGB802w7aWOTJrowUuXGIZbWbWbM8vrtanbRXMEkVOdPT9hVkQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8PR12MB5431
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 10/15/21 4:44 AM, tip-bot2 for Tim Chen wrote:
> The following commit has been merged into the sched/core branch of tip:
> 
> Commit-ID:     66558b730f2533cc2bf2b74d51f5f80b81e2bad0
> Gitweb:        https://git.kernel.org/tip/66558b730f2533cc2bf2b74d51f5f80b81e2bad0
> Author:        Tim Chen <tim.c.chen@linux.intel.com>
> AuthorDate:    Fri, 24 Sep 2021 20:51:04 +12:00
> Committer:     Peter Zijlstra <peterz@infradead.org>
> CommitterDate: Fri, 15 Oct 2021 11:25:16 +02:00
> 
> sched: Add cluster scheduler level for x86
> 
> There are x86 CPU architectures (e.g. Jacobsville) where L2 cahce is
> shared among a cluster of cores instead of being exclusive to one
> single core.
> 
> To prevent oversubscription of L2 cache, load should be balanced
> between such L2 clusters, especially for tasks with no shared data.
> On benchmark such as SPECrate mcf test, this change provides a boost
> to performance especially on medium load system on Jacobsville.  on a
> Jacobsville that has 24 Atom cores, arranged into 6 clusters of 4
> cores each, the benchmark number is as follow:
> 
>   Improvement over baseline kernel for mcf_r
>   copies		run time	base rate
>   1		-0.1%		-0.2%
>   6		25.1%		25.1%
>   12		18.8%		19.0%
>   24		0.3%		0.3%
> 
> So this looks pretty good. In terms of the system's task distribution,
> some pretty bad clumping can be seen for the vanilla kernel without
> the L2 cluster domain for the 6 and 12 copies case. With the extra
> domain for cluster, the load does get evened out between the clusters.
> 
> Note this patch isn't an universal win as spreading isn't necessarily
> a win, particually for those workload who can benefit from packing.
> 
> Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
> Signed-off-by: Barry Song <song.bao.hua@hisilicon.com>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> Link: https://lore.kernel.org/r/20210924085104.44806-4-21cnbao@gmail.com

I've bisected to this patch which now results in my EPYC systems issuing a 
lot of:

[    4.788480] BUG: arch topology borken
[    4.789578]      the SMT domain not a subset of the CLS domain

messages (one for each CPU in the system).

I haven't had a chance to dig deeper and understand everything, does 
anyone have some quick insights/ideas?

Thanks,
Tom