Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 15 Feb 2022 08:44:23 -0800
From:   Darren Hart <darren@os.amperecomputing.com>
To:     Will Deacon <will@kernel.org>
Cc:     "Song Bao Hua (Barry Song)" <song.bao.hua@hisilicon.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Linux Arm <linux-arm-kernel@lists.infradead.org>,
        Catalin Marinas <catalin.marinas@arm.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Valentin Schneider <valentin.schneider@arm.com>,
        "D . Scott Phillips" <scott@os.amperecomputing.com>,
        Ilkka Koskinen <ilkka@os.amperecomputing.com>,
        "stable@vger.kernel.org" <stable@vger.kernel.org>
Subject: Re: [PATCH] arm64: smp: Skip MC domain for SoCs without shared cache
Message-ID: <YgvYZy5xv1g+u5wp@fedora>
References: <8c4a69eca4d0591f30c112df59c5098c24923bd3.1644543449.git.darren@os.amperecomputing.com>
 <ec9be4eb7a0548178191edd51ddd309f@hisilicon.com>
 <20220215163858.GA8458@willie-the-truck>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220215163858.GA8458@willie-the-truck>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?2/X9tEdhgobsD11OspKpxCAnY7T1K2O/5XxDJyuMBMhlMJ10AlE1CMXouoeh?=
 =?us-ascii?Q?zlY5fZX7bFO2zqkiKVdwjqfrhBNkTSiMRslhJDc58F6HnMhrZjZ5Ns9a6XH6?=
 =?us-ascii?Q?g3S3uteO1TrCHDHN2cFyahLwPXlZ8cgaPc44wtRAxeaYfYBuy8l1uCEk352a?=
 =?us-ascii?Q?0+n7ONqIeEKQL+vy94nupXQYVqhFn5wj/leSnRUJtRmnrQH23QR4hmGEtv1k?=
 =?us-ascii?Q?URZGRkEItye9dwCYbCwieyZ9AgZtH2cYFjOlXpS+mtrRthDjUI4IDnx6SVqj?=
 =?us-ascii?Q?t8/MMGDzijUO56nDBVW38lFVrnB2BpCetNNfbktBOhV1dG3TJoqkv+Jqd0dp?=
 =?us-ascii?Q?Gv5upozCLKLjeus1EXFvwZiW++n4hdMy7fztxPm2450wQ4blrusPmjuHZPeK?=
 =?us-ascii?Q?ZdSDCNyVR+bMkXvMtkgA+wduOI0Jls7pYF9Ko+QTDZ6RJia/Qz4GOw/GbYFf?=
 =?us-ascii?Q?f0r09+m1hbR+4wuhVxN/ICxnTyqkeQFCBBEytTHlpCMQT7Cl//CSJtts1M7Z?=
 =?us-ascii?Q?Smv7HU4EIF4QhY4811DT5tAYzA4QUK5TStS0hZKc7Xld+43eZo9CBDWilh5C?=
 =?us-ascii?Q?HKrKocZog5MqMHmOT6OBKMqSJEGHGf/PCxaIUjb/W5m4g1zjk5Ow1uGMvFxt?=
 =?us-ascii?Q?0NtpIswoKTDRaH+g8fygdjYpmWDoTTOILrD9WM6GbmVVrsE+n4FJtavN5Q2o?=
 =?us-ascii?Q?VAlDo5dW5yv7b2QgI/tdSYIiqAerxOt3l4Ao3yljNMjeIADCDaau9qzU47aq?=
 =?us-ascii?Q?kOZSkMiuV1dJjl7ay3u1vaB62ZgFjoySocHrEr+G0zc/HnfKnLjNS8k6q/iD?=
 =?us-ascii?Q?Zy4QOSW+DakA76/Y5kJ9lDq0M91QqYFK7CeUFAOYkeubsWRei1h/npzMQczz?=
 =?us-ascii?Q?tGDKfGxg0id403FW2rXDjoM7SDNMp+VOWwIWsd4DTvuR0JYXYrOG2MJuiad/?=
 =?us-ascii?Q?zPs+/c3zDE0F4dfOGrZqblOJjGCkX8a7c/Gx3Ep/C8niml8YzB4pDxVZ6CQb?=
 =?us-ascii?Q?5kYnQOAgWRjMwtvodg7FYR3pQxOllFuQMzYIt0B8F0r9MXhO7eMQCV8zsK6I?=
 =?us-ascii?Q?ladZJlM9OU/uhKRcFOGVEb0F4ZQVPX6DBMmT/OvapI1sg1021i19wOh0NKA+?=
 =?us-ascii?Q?/aI+SSqwuQkmNeT2MytMTusmggB0Mh7tRHUcpyH3Fb8Ss/qhual79GGAchDe?=
 =?us-ascii?Q?nlLc5EwV8ETAvikvVk9L9b5Qki9bWS/U4t5xzbGitf9NSyqqzuaUw7bIwAW8?=
 =?us-ascii?Q?oqCF89joRqwONxUoLf8gz61inKl0lB93zhtflsfF1lYA334GPZCji0ZDpQxE?=
 =?us-ascii?Q?mB/i5iWgs/FAPyJx3wYJhSztunvHGeWy1/dfi42orkyAuRx4lZj6AAcxWEje?=
 =?us-ascii?Q?t341LlKOO7b4c8/pp8a+Qg+4k5ymB1vZr3eIEh7H+gZXj1bReKi8peujKQPv?=
 =?us-ascii?Q?T4mp2eeoZP4WvVojsRZ1ZTRUISx0VZTT0dgVtbYoFWBon+zwhYoF9UzJ0nr8?=
 =?us-ascii?Q?FHsgJAVEjWBLyUqYbEI+i6N1T0N9EQxkn/XtLoNN0wV4yqpwL/hzKA0MSTlE?=
 =?us-ascii?Q?Cjq4kNl+/LT+Qepothk+E4ukiFAylctKtYCrBJ0FBo+EEJNimhBgqhG12pO/?=
 =?us-ascii?Q?xHqC2ntOFatwry4Px0fNKo9uMSborSA796DMmQ8ldcr0cDBhmXCD6z3oTjdF?=
 =?us-ascii?Q?1lCeTKVLoB2gkXWhtVwZJzfeINa+nRe0CHUvI+bJMiBvAjkuM7lIW32i6TyO?=
 =?us-ascii?Q?4XDbxxp/gw=3D=3D?=
X-OriginatorOrg: os.amperecomputing.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 12d36ac0-b338-477f-76bd-08d9f0a26d4e
X-MS-Exchange-CrossTenant-AuthSource: SA0PR01MB6329.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2022 16:44:26.2143
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: a1Try+TXtFsMJzC7V7N8XQS54IR7kE+qjHygQGHte29xa2ppSHiQPUyJLpvgPyHVfmV1NOtDCmBkqpzRjnSJtWFH+ReHJlDi7IBaMGZ6h95hgxO9fiHrbgrpmMpoUHFy
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR01MB6335
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,
        SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Feb 15, 2022 at 04:38:59PM +0000, Will Decon wrote:
> On Fri, Feb 11, 2022 at 03:20:51AM +0000, Song Bao Hua (Barry Song) wrote:
> > 
> > 
> > > -----Original Message-----
> > > From: Darren Hart [mailto:darren@os.amperecomputing.com]
> > > Sent: Friday, February 11, 2022 2:43 PM
> > > To: LKML <linux-kernel@vger.kernel.org>; Linux Arm
> > > <linux-arm-kernel@lists.infradead.org>
> > > Cc: Catalin Marinas <catalin.marinas@arm.com>; Will Deacon <will@kernel.org>;
> > > Peter Zijlstra <peterz@infradead.org>; Vincent Guittot
> > > <vincent.guittot@linaro.org>; Song Bao Hua (Barry Song)
> > > <song.bao.hua@hisilicon.com>; Valentin Schneider
> > > <valentin.schneider@arm.com>; D . Scott Phillips
> > > <scott@os.amperecomputing.com>; Ilkka Koskinen
> > > <ilkka@os.amperecomputing.com>; stable@vger.kernel.org
> > > Subject: [PATCH] arm64: smp: Skip MC domain for SoCs without shared cache
> > > 
> > > SoCs such as the Ampere Altra define clusters but have no shared
> > > processor-side cache. As of v5.16 with CONFIG_SCHED_CLUSTER and
> > > CONFIG_SCHED_MC, build_sched_domain() will BUG() with:
> > > 
> > > BUG: arch topology borken
> > >      the CLS domain not a subset of the MC domain
> > > 
> > > for each CPU (160 times for a 2 socket 80 core Altra system). The MC
> > > level cpu mask is then extended to that of the CLS child, and is later
> > > removed entirely as redundant.
> > > 
> > > This change detects when all cpu_coregroup_mask weights=1 and uses an
> > > alternative sched_domain_topology equivalent to the default if
> > > CONFIG_SCHED_MC were disabled.
> > > 
> > > The final resulting sched domain topology is unchanged with or without
> > > CONFIG_SCHED_CLUSTER, and the BUG is avoided:
> > > 
> > > For CPU0:
> > > 
> > > With CLS:
> > > CLS  [0-1]
> > > DIE  [0-79]
> > > NUMA [0-159]
> > > 
> > > Without CLS:
> > > DIE  [0-79]
> > > NUMA [0-159]
> > > 
> > > Cc: Catalin Marinas <catalin.marinas@arm.com>
> > > Cc: Will Deacon <will@kernel.org>
> > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > Cc: Vincent Guittot <vincent.guittot@linaro.org>
> > > Cc: Barry Song <song.bao.hua@hisilicon.com>
> > > Cc: Valentin Schneider <valentin.schneider@arm.com>
> > > Cc: D. Scott Phillips <scott@os.amperecomputing.com>
> > > Cc: Ilkka Koskinen <ilkka@os.amperecomputing.com>
> > > Cc: <stable@vger.kernel.org> # 5.16.x
> > > Signed-off-by: Darren Hart <darren@os.amperecomputing.com>
> > 
> > Hi Darrent,
> > What kind of resources are clusters sharing on Ampere Altra?
> > So on Altra, cpus are not sharing LLC? Each LLC is separate
> > for each cpu?
> > 
> > > ---
> > >  arch/arm64/kernel/smp.c | 32 ++++++++++++++++++++++++++++++++
> > >  1 file changed, 32 insertions(+)
> > > 
> > > diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> > > index 27df5c1e6baa..0a78ac5c8830 100644
> > > --- a/arch/arm64/kernel/smp.c
> > > +++ b/arch/arm64/kernel/smp.c
> > > @@ -715,9 +715,22 @@ void __init smp_init_cpus(void)
> > >  	}
> > >  }
> > > 
> > > +static struct sched_domain_topology_level arm64_no_mc_topology[] = {
> > > +#ifdef CONFIG_SCHED_SMT
> > > +	{ cpu_smt_mask, cpu_smt_flags, SD_INIT_NAME(SMT) },
> > > +#endif
> > > +
> > > +#ifdef CONFIG_SCHED_CLUSTER
> > > +	{ cpu_clustergroup_mask, cpu_cluster_flags, SD_INIT_NAME(CLS) },
> > > +#endif
> > > +	{ cpu_cpu_mask, SD_INIT_NAME(DIE) },
> > > +	{ NULL, },
> > > +};
> > > +
> > >  void __init smp_prepare_cpus(unsigned int max_cpus)
> > >  {
> > >  	const struct cpu_operations *ops;
> > > +	bool use_no_mc_topology = true;
> > >  	int err;
> > >  	unsigned int cpu;
> > >  	unsigned int this_cpu;
> > > @@ -758,6 +771,25 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
> > > 
> > >  		set_cpu_present(cpu, true);
> > >  		numa_store_cpu_info(cpu);
> > > +
> > > +		/*
> > > +		 * Only use no_mc topology if all cpu_coregroup_mask weights=1
> > > +		 */
> > > +		if (cpumask_weight(cpu_coregroup_mask(cpu)) > 1)
> > > +			use_no_mc_topology = false;
> > 
> > This seems to be wrong? If you have 5 cpus,
> > Cpu0 has cpu_coregroup_mask(cpu)== 1, cpu1-4
> > has cpu_coregroup_mask(cpu)== 4, for cpu0, you still
> > need to remove MC, but for cpu1-4, you will need
> > CLS and MC both?
> 
> What is the *current* behaviour on such a system?
> 

As I understand it, any system that uses the default topology which has
a cpus_coregroup weight of 1 and a child (cluster, smt, ...) weight > 1
will behave as described above by printing the following for each CPU
matching this criteria:

  BUG: arch topology borken
        the [CLS,SMT,...] domain not a subset of the MC domain

And then extend the MC domain cpumask to match that of the child and continue
on.

That would still be the behavior for this type of system after this
patch is applied.

Thanks,

-- 
Darren Hart
Ampere Computing / OS and Kernel