Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
From:   Qing Wang <wangqing@vivo.com>
To:     Catalin Marinas <catalin.marinas@arm.com>,
        Will Deacon <will@kernel.org>,
        Sudeep Holla <sudeep.holla@arm.com>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        "Rafael J. Wysocki" <rafael@kernel.org>,
        linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org
Cc:     vincent.guittot@linaro.org, dietmar.eggemann@arm.com,
        Wang Qing <wangqing@vivo.com>
Subject: [PATCH V2 2/2] arm64: Add complex scheduler level for arm64
Date:   Fri, 22 Apr 2022 04:51:26 -0700
Message-Id: <1650628289-67716-3-git-send-email-wangqing@vivo.com>
In-Reply-To: <1650628289-67716-1-git-send-email-wangqing@vivo.com>
References: <1650628289-67716-1-git-send-email-wangqing@vivo.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RUZ2WEhBb1liTFJyM29Bcko0Y1lIRXlqR0RoS0FYSDhFSG5lUVJIRnVhSEZj?=
 =?utf-8?B?bWVuQjhsOUFaSHRYRk9kbWtpUVl5aGR0Z0xHUEpoZW4wNWtPWEVWM1lzK2tv?=
 =?utf-8?B?MFlMenBWTjNIeDY4djZLNno5b0gxREdBZWpFb2ZxdXpvbVhLV3Nramd4dFI1?=
 =?utf-8?B?aEJHelREcXN1WThtUVpEOUxzeHg3ME1kelB6Um1XSENkSEQ4THprc0wyWXJC?=
 =?utf-8?B?QWFBOFBLd0NEY0c4L2srakFTbXBybzFHU0ZtdWk0Vkx1T280dFVaeEp2ak9t?=
 =?utf-8?B?dE1KdkI5bFgrQ29EMDFBc0VyQkhKT1h1d21RU3N0YTQycWx0NVJhUnVGaGQ3?=
 =?utf-8?B?bEtZMFl0ZHZxM2l3L3JLUWNHRlNSWnBJVVNFNFZ2WG9UWWk3dE5YZyt6TmNE?=
 =?utf-8?B?SUtwUllmN2lNOThvbGJwY3czUlFrb2E2TzROUEtGUkxWb3d0M1dlZUZnc2cr?=
 =?utf-8?B?WWdTQnFXQkxDdU1iODkvWlVEYSs1YTVYQjlLOGsxUWRnanhFMy8xM2ZVRXBH?=
 =?utf-8?B?enFHcE1GSGU4ZU96S3ArWWhBbXA2dTFZRCt6K3dvL3Vqb252Q1MwTWFpdFIv?=
 =?utf-8?B?QUl6dTBxREc4WEZ3cXdWUC9HWmFMR25JaFU0MnlUUGZJVFJKMEJyOFlBWDQ5?=
 =?utf-8?B?QUFzWmtDaUdrb1lYcE9MQkZvNVRsVGpucDM0dzJTcUNiT3RHMlh3QWJnTXlG?=
 =?utf-8?B?bFJNTytiNURiR2U3dWtOblp5V1d1aDJCUHNsVFJwY1NmdDk4S1N0Q1RtSE5J?=
 =?utf-8?B?YktGbFFhMzM3Tjl3eGlTUThJKzdsWDg4bk5taEN0M2RoYXRaVXpBYTY0WE95?=
 =?utf-8?B?am5jcTVRNnhyWU5uYzJtb1BSOTVLaDliYk1RL1VHQXdXU3BYeXlBZFpyUFl4?=
 =?utf-8?B?aFllUnQzckh4MkRUejRpWXd3QjFXS014OXN2aUhRZ1lxU0JsNzlGMmdvcGtt?=
 =?utf-8?B?RHJyZVMrVGJ6QnZ5bHNCYUxNMEpqMWFacUttNTJtWHBiU1BPV3ZtanR3R0hS?=
 =?utf-8?B?M3NSUFJsNms4RGJINTJ4VUJoaEpSNlZEbVVLaWc5dHhMOVh3MjlQRXYyNTYw?=
 =?utf-8?B?WUZXQytiWloyV0ZLVnhOVWgyTXZCUmErK0lVV05DZFpXM2txNWltZEl3QXZV?=
 =?utf-8?B?RjFWYkpyd29MNHVmS3duRndwV0ltVHRZV2tkaTlFaHRpY3k4OFloaVhxSnBx?=
 =?utf-8?B?U2tYaGJVZ1JMV3RTQ0h6VFl1UncyVktnSnVTNUhYMWpnZncrVnUyUzBHOGVo?=
 =?utf-8?B?VXBSYjRkSDJrelhwbHh5QjViQzNKU1dzWElyZmNHbGdYdGJNSTJDQW1rZWht?=
 =?utf-8?B?V0Zwdzk2TUpXV2tKamF3Tk54N0pEZ1BmZjBIQ2tTTCtxSDFmVWJpeVVjY1dR?=
 =?utf-8?B?dlZWWHNGRzhiTDdDUmV2UWZNTXA2WHdhS2Yxc2NqNHFGVGIrLzBxck5uTysr?=
 =?utf-8?B?NU5ZaFRJUERweG55eG5ieVhEdXlzem96NkZnQ1BCSjJSSVdXVUo2SWhHVXUr?=
 =?utf-8?B?S1lxcUR6bUtFRm1ZY1F6enJtVmhhYVJxMm1Hc1pWUTFFT0pkNS8vUHduSkZU?=
 =?utf-8?B?WXArSzVoMnRrbENaMDdEbEVrbnFVSXd4UzJGU1VlU0x4c3ZZYnZ5alNaZTJN?=
 =?utf-8?B?bXo1ZFJHdm1sdnpoRzBpdkN4YytmdGxyRHpUTmRUZmRWd3A2aG12N0YwMm1P?=
 =?utf-8?B?NFdmOVBURmRjcEgwQUxrOE95TEo4SVFhc3JzR1BENHNFa0k4T3JZV2prUXlG?=
 =?utf-8?B?U1Iyc1l2SDVUdmlHcnEzQUoxZmhjOGhUUGRjNWIwNXladDRqVG1NRllWcXox?=
 =?utf-8?B?aExUQjluSmhoWEsyeE5tZU5VTGhKdFZmZ0RQbU54VmY4eHNTTk4xSURRTnZI?=
 =?utf-8?B?cWhRcUxXWm8yZVNIRTZJN1l0bjZIWUQwOE52dlJYZHVhdU1IZDdFK0JWSGhJ?=
 =?utf-8?B?citCaGw3dWdid25kWVlIalNLRjFBWXdGNjk4cE1VMkNqWGVQRHdWdGVQM1cy?=
 =?utf-8?B?UjVvLzFPZDROcUk5QXlTUHhyaHBDb0NKL012VjFwakVmTFczSUtIejVyQkVK?=
 =?utf-8?B?L0VFR1Q0T09nekdiNlY1cEdJcGZSWmk3ZDRXcDF0NlpxWDFFSjNRWkZWKzdI?=
 =?utf-8?B?eDVrWVRsVE9kQ0h2d0l1Q3ZxSzhYZ0xTbFFmTGp5RHduS1lBQm9HU0RDUCt3?=
 =?utf-8?B?MHJUSFhFUm1ZVmgxT0ZUYURQTWlDTzNvQWNUdGRNbDR6WS94NkVIRHV1c1pv?=
 =?utf-8?B?bml4WEwvb2tvUHE5bmp4Q0RmY2lGd0I5VmJDRU9jYnc3T1ViaUs3Y0NCZkJR?=
 =?utf-8?B?ajIzUXVpVjBXYStEbDkrMENrK01JN1kreERkazZzREV2L29WVUQzOU56MXNX?=
 =?utf-8?Q?4Zf9CMExVEUInZEY=3D?=
X-OriginatorOrg: vivo.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9d699879-2e87-4249-7fb8-08da24568446
X-MS-Exchange-CrossTenant-AuthSource: SL2PR06MB3082.apcprd06.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Apr 2022 11:52:03.3065
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 923e42dc-48d5-4cbe-b582-1a797a6412ed
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: YRR9H59bvSvLrbKkZwwdqDYkqlKFvaWVWudefc+eY9zQAWYKmDdpwYFAV8bJJ524CIbSf4Zii8P/qPvVD3QNrA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SG2PR06MB3726
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RDNS_NONE,
	SPF_HELO_NONE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

From: Wang Qing <wangqing@vivo.com>

The DSU-110 DynamIQ™ cluster supports blocks that are called complexes
which contain up to two cores of the same type and some shared logic.
Sharing some logic between the cores can make a complex area efficient.

This patch adds complex level for complexs and automatically enables
the load balance among complexs. It will directly benefit a lot of
workload which loves more resources such as memory bandwidth, caches.

Testing has been done with Stream benchmark:
8threads stream (2 little cores * 2(complex) + 3 medium cores + 1 big core)
                stream                 stream
                w/o patch              w/ patch
MB/sec copy     37579.2 (   0.00%)    39127.3 (   4.12%)
MB/sec scale    38261.1 (   0.00%)    39195.4 (   2.44%)
MB/sec add      39497.0 (   0.00%)    41101.5 (   4.06%)
MB/sec triad    39885.6 (   0.00%)    40772.7 (   2.22%)

And in order to support this features, we defined arm64_topology.

V2:
fix commit log and loop more

Signed-off-by: Wang Qing <wangqing@vivo.com>
---
 arch/arm64/Kconfig      | 13 +++++++++++
 arch/arm64/kernel/smp.c | 48 ++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 60 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index edbe035cb0e3..4063de8c6153 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1207,6 +1207,19 @@ config SCHED_CLUSTER
 	  by sharing mid-level caches, last-level cache tags or internal
 	  busses.
 
+config SCHED_COMPLEX
+	bool "Complex scheduler support"
+	help
+	  DSU supports blocks that are called complexes which contain up to
+	  two cores of the same type and some shared logic. Sharing some logic
+	  between the cores can make a complex area efficient.
+
+	  Complex also can be considered as a shared cache group smaller
+	  than cluster.
+
+	  Complex scheduler support improves the CPU scheduler's decision
+	  making when dealing with machines that have complexs of CPUs.
+
 config SCHED_SMT
 	bool "SMT scheduler support"
 	help
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 3b46041f2b97..526765112146 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -14,6 +14,7 @@
 #include <linux/sched/mm.h>
 #include <linux/sched/hotplug.h>
 #include <linux/sched/task_stack.h>
+#include <linux/sched/topology.h>
 #include <linux/interrupt.h>
 #include <linux/cache.h>
 #include <linux/profile.h>
@@ -57,6 +58,10 @@
 DEFINE_PER_CPU_READ_MOSTLY(int, cpu_number);
 EXPORT_PER_CPU_SYMBOL(cpu_number);
 
+#ifdef SCHED_COMPLEX
+DEFINE_PER_CPU_READ_MOSTLY(cpumask_t, cpu_complex_map);
+#endif
+
 /*
  * as from 2.5, kernels no longer have an init_tasks structure
  * so we need some other way of telling a new secondary core
@@ -715,6 +720,47 @@ void __init smp_init_cpus(void)
 	}
 }
 
+#ifdef SCHED_COMPLEX
+static int arm64_complex_flags(void)
+{
+	return SD_SHARE_PKG_RESOURCES;
+}
+
+const struct cpumask *arm64_complex_mask(int cpu)
+{
+	const struct cpumask *core_mask = cpu_cpu_mask(cpu);
+
+	/* Find the smaller shared cache level than clustergroup and coregroup*/
+#ifdef CONFIG_SCHED_MC
+	core_mask = cpu_coregroup_mask(cpu);
+#endif
+#ifdef CONFIG_SCHED_CLUSTER
+	core_mask = cpu_clustergroup_mask(cpu);
+#endif
+
+	find_subset_of_share_cache(core_mask, cpu, &per_cpu(cpu_complex_map, cpu));
+
+	return &per_cpu(cpu_complex_map, cpu);
+}
+#endif
+
+static struct sched_domain_topology_level arm64_topology[] = {
+#ifdef CONFIG_SCHED_SMT
+	{ cpu_smt_mask, cpu_smt_flags, SD_INIT_NAME(SMT) },
+#endif
+#ifdef CONFIG_SCHED_COMPLEX
+	{ arm64_complex_mask, arm64_complex_flags, SD_INIT_NAME(CPL) },
+#endif
+#ifdef CONFIG_SCHED_CLUSTER
+	{ cpu_clustergroup_mask, cpu_cluster_flags, SD_INIT_NAME(CLS) },
+#endif
+#ifdef CONFIG_SCHED_MC
+	{ cpu_coregroup_mask, cpu_core_flags, SD_INIT_NAME(MC) },
+#endif
+	{ cpu_cpu_mask, SD_INIT_NAME(DIE) },
+	{ NULL, },
+};
+
 void __init smp_prepare_cpus(unsigned int max_cpus)
 {
 	const struct cpu_operations *ops;
@@ -723,9 +769,9 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
 	unsigned int this_cpu;
 
 	init_cpu_topology();
-
 	this_cpu = smp_processor_id();
 	store_cpu_topology(this_cpu);
+	set_sched_topology(arm64_topology);
 	numa_store_cpu_info(this_cpu);
 	numa_add_cpu(this_cpu);
 
-- 
2.7.4