Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Date:   Wed, 23 Feb 2022 08:39:26 -0800
From:   Darren Hart <darren@os.amperecomputing.com>
To:     Vincent Guittot <vincent.guittot@linaro.org>
Cc:     Will Deacon <will@kernel.org>,
        "Song Bao Hua (Barry Song)" <song.bao.hua@hisilicon.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Linux Arm <linux-arm-kernel@lists.infradead.org>,
        Catalin Marinas <Catalin.Marinas@arm.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Valentin Schneider <Valentin.Schneider@arm.com>,
        "D . Scott Phillips" <scott@os.amperecomputing.com>,
        Ilkka Koskinen <ilkka@os.amperecomputing.com>,
        "stable@vger.kernel.org" <stable@vger.kernel.org>
Subject: Re: [PATCH] arm64: smp: Skip MC domain for SoCs without shared cache
Message-ID: <YhZjPviteu4v8Fdf@fedora>
References: <20220215164639.GC8458@willie-the-truck>
 <CAKfTPtAFsL+uqQiGcuh+JJZB=rPrez0=kotq76CVRcBQhcPefg@mail.gmail.com>
 <YgvjtsOZuxzbgBBl@fedora>
 <CAKfTPtCHrZCp1Uth4odyT721wE8zoNVY3Mh+DSyuTkqPafm0Hg@mail.gmail.com>
 <YgwHhxy/uGafQsak@fedora>
 <CAKfTPtAR2+bY8QpyaCCJfezsVkB62n8XZjL9c5_mPO3iyDnp4w@mail.gmail.com>
 <Yg0lULy5TmHKIHFv@fedora>
 <CAKfTPtB1Vt75ciX_V=8T3e5fgW-X7ybRk6VZvy4uXzjazjx9ZA@mail.gmail.com>
 <YhWlDUzFeG0d7z6C@fedora>
 <CAKfTPtAjnRGc9c1Ni0ru6Xz9wKLPoBY4wdPkN0uFBzR-_iurPg@mail.gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAKfTPtAjnRGc9c1Ni0ru6Xz9wKLPoBY4wdPkN0uFBzR-_iurPg@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?HKlF59XE3twX0pTi3/5X+8/zaitpX25EFmc0R06KwtxXvi/wJ9YqHQES+3nk?=
 =?us-ascii?Q?v+bozIIR1IYlig7vadOG0K03qdDhHomYRUo30r5rsrYs6X69A0hF7ok0jrMQ?=
 =?us-ascii?Q?C/q5ZtXNMhU7H70rn+2NHgGwQEp405SV3e+3xC100yVGBxd9oYQV/XP3YX2C?=
 =?us-ascii?Q?buTBVZLFBNLfPi79LlqTNU7PfBNeeeMEf2rSQtdj6B+e827hRoachlXsHol5?=
 =?us-ascii?Q?SfpeyCLrOsVZsK6GGz4hQnPYYfU81HOnf3QcZn7cSK0jHNX13BXbHfBTR+Hx?=
 =?us-ascii?Q?3jhnvH/WDbRVLPikKNOImakcjlMVzYWOPLFTItHek1vdrnTEoSd5msOmq6+e?=
 =?us-ascii?Q?7NkOH3W5I1tEKm3fA/52KMY8oUltbwYUeZUsJTWtnyt/U4XBJVm/tiIsztYT?=
 =?us-ascii?Q?ahK1lbbFAUT7Yx3xiJX4l0y5vCzscshbB44YEFU3f0D855eK63mWfcqk4AHO?=
 =?us-ascii?Q?FcP/mOXoF9irzSUHwFlOGF3hnUnIs5LiXhlmVANTguPq2MjkoFPGTJZRuywW?=
 =?us-ascii?Q?9CA+sPymQ0CYDC1c+GriXJhpEetw5pH43pmUTaDcUyxg+7bfmo4DB1skMSCi?=
 =?us-ascii?Q?S3rejp6lxOLxPVNNMGK+f84urT5PbjTKUcT5npmlurAlSDOlPxvKjjT3n/nN?=
 =?us-ascii?Q?7AfaJEN0W0DYGgVh2n7ztSqTU6iPLt1RS8tsA1NYhjdEVcoPPPnF+nFYslmJ?=
 =?us-ascii?Q?ZafM/Ic7ozDFVEmbEK1undu8N1zEXj6olWWG0IJYmWtHIzJm5FNXtKTTFJE9?=
 =?us-ascii?Q?6X1Qj3tVZxnO66nQlTA0VVC/XTMpsCXDM9UFb1GRPrqtJGzex+UDohiaPtcI?=
 =?us-ascii?Q?x2tRA3Ga1MTFd1mNb0iD843KeuUCsgpL1i0xSLmXvKH1sFIBZPkcafdtCNv8?=
 =?us-ascii?Q?BbxN/cS3YSlgjNWDp3r8i/6hdZrqwabN+FslF+GsL/E9+O07/Ck3hTLvj7G2?=
 =?us-ascii?Q?1qcgfK9E9ll2gyCwT9sEHN+1J1z2sowp0TCM2qHjlZg5sRLy3biNhH5D3h7W?=
 =?us-ascii?Q?H78F7p9kzFPhcp4s0THwJtkKHHtJDGXpn++glQYCnvMVJ+XUU154KK0UHkwi?=
 =?us-ascii?Q?yf9grp6d5dKFiS5pnE3Hja2BB0q+k0exqFY5kSEz4a8lqwuIgybiAK6TX2oj?=
 =?us-ascii?Q?8k9xxYVu47WPwSvTC3yMKhVETHvcMv3+t44F9sR1b36kGRH8zeJXOTqjOfNU?=
 =?us-ascii?Q?/sC0DjTcGM86uur6pmRFiV3pBGit2HGE8G9dXyRxnQIdrl9eYeWouYYJyr+J?=
 =?us-ascii?Q?P5IZhwPKb5H2TctjzCL7dyZAukENb2hqBxaTlownZt66vCbZAzr+abXUHRRw?=
 =?us-ascii?Q?fQH1dMKI8Qsy1+G60yE6T6tyfSjgzQ/GfrucathlMJzNmiML1VuzVIJCkL50?=
 =?us-ascii?Q?z7rTj4inOFK4b08IoUsLe5Iw+1g7ycs19deUmsugUeMCqUeO74Vjm+49OR6u?=
 =?us-ascii?Q?+pvzIlaHZwEx9I29u9J/HKgHzSlGvjk9grZEe36ocqCz1mwvCmk96EDkpAfN?=
 =?us-ascii?Q?7UVFsWLJiyNE8SMLXaEH8oWukcrgvvR+Z0gcO27CaYV1vC9bW9KQJcPL88wh?=
 =?us-ascii?Q?H5P15KxyiqMY3gD+7GEBbeOlMtH4RfUVASTWWFlE1O7bgWEFluf4KMCMl5PE?=
 =?us-ascii?Q?Wi12SJts/WhiATyXtPdvfCdgOk9YGFGXUwryKPFjQHJ/n/hEHDdEsqQGn5Ih?=
 =?us-ascii?Q?1vOIoruiFeCH8m9Uq8D7A32CJvk55nH7lXch17q1VsTPXZ3+iGWK9R5sxbum?=
 =?us-ascii?Q?zIXePR9Jdw=3D=3D?=
X-OriginatorOrg: os.amperecomputing.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d2335ccc-479c-4ea7-2869-08d9f6eb0f0a
X-MS-Exchange-CrossTenant-AuthSource: SA0PR01MB6329.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2022 16:39:29.0185
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: zKd8xSvKrvwi1+1ZK5YRy0RXQoLOglZZ3KzG8xoHGjy3QbLMRArVcnaTO11M9Or1Nh6RRM5IH1aOhnx8e8q/QFnG7xKxzl9IRfETPxlm6R7zjNeRljlYOz3IabXYOA5T
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO2PR01MB2118
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Feb 23, 2022 at 09:19:17AM +0100, Vincent Guittot wrote:

...

> > > AFAICT, this CLUSTER level is only supported by ACPI. In
> > > parse_acpi_topology() you should be able to know if cluster level is
> > > above or below the level returned by acpi_find_last_cache_level() and
> > > set the correct topology table accordingly
> > >
> >
> > Thanks Vincent,
> >
> > This made sense as a place to start to me. The more I dug into the ACPI PPTT
> > code, I kept running into conflicts with the API which would require extending
> > it in ways that seems contrary to its intent. e.g. the exposed API uses Kernel
> > logical CPUs rather than the topology ids (needed for working with processor
> > containers). The cpu_topology masks haven't been populated yet, and
> > acpi_find_last_cache_level is decoupled from the CPU topology level. So what
> > we're really testing for is if the cluster cpumask is a subset of the coregroup
> > cpumask or not, and it made the most sense to me to keep that in smp.c after the
> > cpumasks have been updated and stored.
> 
> I'm not sure why you want to compare cpumask when you can directly
> compare topology level which is exactly what we are looking for in
> order to correctly order the scheduler topology. I was expecting
> something like the below to be enough. acpi_find_cluster_level() needs
> to be created and should be similar to
> find_acpi_cpu_topology_cluster() but return level instead of id. The
> main advantage is that everything is contained in topology.c which
> makes sense as we are playing with topology

Hi Vincent,

This was my thinking as well before I dug into the acpi pptt interfaces.

The cpu topology levels and the cache levels are independent and assuming I've
not misunderstood the implementation, acpi_find_cache_level() returns the
highest *cache* level described in the PPTT for a given CPU.

For the Ampere Altra, for example:

CPU Topo  1       System
 Level    2         Package
   |      3           Cluster
   |      4             Processor --- L1 I Cache \____ L2 U Cache -x
  \/                              --- L1 D Cache /

          4             Processor --- L1 I Cache \____ L2 U Cache -x
                                  --- L1 D Cache /

                  Cache Level ---->    1                2

acpi_find_cache_level() returns "2" for any logical cpu, but this doesn't tell
us anything about the CPU topology level across which this cache is shared.

This is what drove me out of topology.c and up into smp.c after the various
topologies are populated and comparing masks. I'll spend a bit more time before
sending a cpumask implementation to see if there is a better point to do this
where the cpu topology level with shared cache is more readily available.

> 
> diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c
> index 9ab78ad826e2..4dac0491b7e3 100644
> --- a/arch/arm64/kernel/topology.c
> +++ b/arch/arm64/kernel/topology.c
> @@ -84,6 +84,7 @@ static bool __init acpi_cpu_is_threaded(int cpu)
>  int __init parse_acpi_topology(void)
>  {
>         int cpu, topology_id;
> +       bool default_cluster_topology = true;
> 
>         if (acpi_disabled)
>                 return 0;
> @@ -119,8 +120,16 @@ int __init parse_acpi_topology(void)
>                         if (cache_id > 0)
>                                 cpu_topology[cpu].llc_id = cache_id;
>                 }
> +
> +               if (default_cluster_topology &&
> +                   (i < acpi_find_cluster_level(cpu))) {

Per above, from what I understand, this is comparing cpu topology levels with
cache levels, which are independent from each other.

> +                       default_cluster_topology = false;
> +               }
>         }
> 
> +       if (!default_cluster_topology)
> +               set_sched_topology(arm64_no_mc_topology);
> +
>         return 0;
>  }

Thanks,

-- 
Darren Hart
Ampere Computing / OS and Kernel