Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 22 Feb 2022 19:20:12 -0800
From:   Darren Hart <darren@os.amperecomputing.com>
To:     Barry Song <21cnbao@gmail.com>
Cc:     Vincent Guittot <vincent.guittot@linaro.org>,
        Will Deacon <will@kernel.org>,
        "Song Bao Hua (Barry Song)" <song.bao.hua@hisilicon.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Linux Arm <linux-arm-kernel@lists.infradead.org>,
        Catalin Marinas <Catalin.Marinas@arm.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Valentin Schneider <Valentin.Schneider@arm.com>,
        "D . Scott Phillips" <scott@os.amperecomputing.com>,
        Ilkka Koskinen <ilkka@os.amperecomputing.com>,
        "stable@vger.kernel.org" <stable@vger.kernel.org>
Subject: Re: [PATCH] arm64: smp: Skip MC domain for SoCs without shared cache
Message-ID: <YhWn7MmBvgZzP7CA@fedora>
References: <ec9be4eb7a0548178191edd51ddd309f@hisilicon.com>
 <20220215163858.GA8458@willie-the-truck>
 <YgvYZy5xv1g+u5wp@fedora>
 <20220215164639.GC8458@willie-the-truck>
 <CAKfTPtAFsL+uqQiGcuh+JJZB=rPrez0=kotq76CVRcBQhcPefg@mail.gmail.com>
 <YgvjtsOZuxzbgBBl@fedora>
 <CAKfTPtCHrZCp1Uth4odyT721wE8zoNVY3Mh+DSyuTkqPafm0Hg@mail.gmail.com>
 <CAGsJ_4yB-FOPoPjCn+T4m76tvzA6ATaz24KYM9NjBeB54nWxLA@mail.gmail.com>
 <Yg0bu53iACDscIC6@fedora>
 <CAGsJ_4w2r8Hp3BNOrcQYDT6JgsFWWAgVruAOXpeXrhjskJMV7w@mail.gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAGsJ_4w2r8Hp3BNOrcQYDT6JgsFWWAgVruAOXpeXrhjskJMV7w@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?nNgrt6xJl4gFc4tDPKJ56JSnhJweeP5YlJSU1EfkjqUL2bvAmSUp8E3UoHIs?=
 =?us-ascii?Q?7Ah5HSpdeZsCWF+mdvQC6/PpJTZpwuHngOREiC89d7zWq2FVStbp/Ll7lwOl?=
 =?us-ascii?Q?oqcWefNHuUmPloiKc2dD1DTNzB80+hxPK+udn8f1Y1JIpHgqWo1YJT8yu2e5?=
 =?us-ascii?Q?S62Q+NNISuJ4v6OOUPJ3t8auqsNpGmyzAF5NN+v7VHmFkDCrc3aVR0Hy49xV?=
 =?us-ascii?Q?9DHnAYhpvKdfAfcqLgd1AOtMUOVzRhs1QkP36jTQEbPCS/bpz7iWrPDMU91K?=
 =?us-ascii?Q?vfOl62A7VbGYkISgP65zJeET3yfBHcO+7FdzFXJmkKtq99LHvt53SGHz4Dc2?=
 =?us-ascii?Q?ms+e8paLqdr/uksu2XKXO6D4AZgcEdNRN5MMd0zC4m38I5GK/cjkuSKmyZF+?=
 =?us-ascii?Q?bMVSJxTQddC2phG5OE2SPbiNN2YyD+IYDALGX0dADyPsyBncqxLkKWK602Rt?=
 =?us-ascii?Q?+lfdf+e/5GVr+mBo6zEpqdB63qPXQklUvfnvpdKtX6vyvT0+X9SsyXf4fQlx?=
 =?us-ascii?Q?WdAluu4bq/gVqi6AERrcMPs8krmebbqVFAqaCzT/jH161kKEm0IEw+dTugrA?=
 =?us-ascii?Q?hCMF7a9PhOCQ7q2KY/yN5JHwJaQzeVTF9FL9cWuLJHrDlFdzApUezCEAT1Ds?=
 =?us-ascii?Q?Km+g8Pk9V0t2C4BFVl5IJn+4RQX1togh/i6pSxntZ1aOvmjhY/f0iqbiM0D7?=
 =?us-ascii?Q?9M4SyyJQvobm7rYGVtLFM3bBdBym4enYr4zNPXK7YgxSetym01K31YVDjxyT?=
 =?us-ascii?Q?vsCXmz8lJ+YrFmg2vqtOSw8LnRye/SStWTJt4kB+kfYP2Q5eJG4rFfBPsV/1?=
 =?us-ascii?Q?zX5htBmJtUK4TX6d1sHJECOsXX1BErWHpYqvKNAK/FOyetfuHoF+lpyjJZtO?=
 =?us-ascii?Q?nPBKNTLt2VaCtRMNWhXjpUjd9/xF4Ibiv0XuqwrJBJqxh6fxMtiU+OO57x+b?=
 =?us-ascii?Q?nYG2sXFN1hzuyzYQxuXfqy+JrU5PDgMWy2eVlHUd2YbnMHRn3M83AY/n35qO?=
 =?us-ascii?Q?6j2nY4irTS64IqSE9bpOtDRCmq96V5z0R2YLyvxd4eWe8KUKXfwdeuV43nhU?=
 =?us-ascii?Q?rHlx+Qdw8vafy8Au95YGRme3hwfkm8/E+46tAzIi757hoCyTyYIYB+dL7yeB?=
 =?us-ascii?Q?y7svrGY6EZbvwCzxvwxSIMJqkgRk82UqfhI1MTGEoIsTVwEsbW5UvEBAPBZy?=
 =?us-ascii?Q?ApsveD8O7sCslmZsgKi/2wuVw6AWlhDIcDvxPylrcHDUi3qCyk+QT5ZcGho/?=
 =?us-ascii?Q?1e7pQeKbxJLM1IswAjbdzZyHXh7YKHDREZO6iLlqlnvTEQWUowY+CTFThkzN?=
 =?us-ascii?Q?5TkOdQ4Fz49Y3+QgS38uwys1O92k3erzxDrXa76vREV9vAs2y03iPypesXZp?=
 =?us-ascii?Q?7eMIcAXqRxoxoNVr23MhKZlXPaLD7tAnISUul9CH+1O41kmYpjGHo88K4b/Z?=
 =?us-ascii?Q?FGeMts9vH49tpA8VehOKP8ndvqU5+5RXrUXw0T1J51bIK7Ct+6emp9pT2FVD?=
 =?us-ascii?Q?pR4hPvT9xTOIls6dzXYLrx4sr94y5tiZsK60pCa5/QN0LzDf5qqb07FrZBjL?=
 =?us-ascii?Q?bWnxcNHGGToukevCN/JBSlkjYtz6g7yHlIq8ix4Md3Mcvnp3/EVga3c7BN2a?=
 =?us-ascii?Q?2yq9Mfu+WJx6YxsgeddlVuPR6/QkCAt/a/79I4uQaoCrkTKbrgD8zFAtHDzI?=
 =?us-ascii?Q?XiAIRAIuSujQXQWt2r8FKNnf2p1ks8rbyi3K7XgHC0tTvac/M0Cf3DCF6V7c?=
 =?us-ascii?Q?kxXAY3jPZw=3D=3D?=
X-OriginatorOrg: os.amperecomputing.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5090c48a-954f-4363-9cc1-08d9f67b68e3
X-MS-Exchange-CrossTenant-AuthSource: SA0PR01MB6329.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2022 03:20:15.2199
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: MrIy3WnyI3z8QU5z0VsPQlFJp3CDMtiuEQRZRxje/N7IWdm8+dm4VXgcB7kX25BJnSuzGuIB6qP3nNS73oTy76cD7jn12x01wEA95QMxSu1rDslv8G+mFejgZWkC0Aht
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR01MB6294
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,
        SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Feb 17, 2022 at 07:56:00AM +1300, Barry Song wrote:
...
> > > > Then, there is another point:
> > > > In your case, CLUSTER level still has the flag SD_SHARE_PKG_RESOURCES
> > > > which is used to define some scheduler internal variable like
> > > > sd_llc(sched domain last level of cache) which allows fast task
> > > > migration between this cpus in this level at wakeup. In your case the
> > > > sd_llc should not be the cluster but the MC with only one CPU. But I
> > > > would not be surprised that most of perf improvement comes from this
> > > > sd_llc wrongly set to cluster instead of the single CPU
> > >
> > > I assume this "mistake" is actually what Ampere altra needs while it
> > > is wrong but getting
> > > right result? Ampere altra has already got both:
> >
> > Hi Barry,
> >
> > Generally yes - although I do think we're placing too much emphasis on
> > the "right" or "wrong" of a heuristic which are more fluid in
> > definition over time. (e.g. I expect this will look different in a year
> > based on what we learn from this and other non current default topologies).
> >
> > > 1. Load Balance between clusters
> > > 2. wake_affine by select sibling cpu which is sharing SCU
> > >
> > > I am not sure how much 1 and 2 are helping Darren's workloads respectively.
> >
> > We definitely see improvements with load balancing between clusters.
> > We're running some tests with the wake_affine patchset you pointed me to
> > (thanks for that). My initial tbench runs resulted in higher average and
> > max latencies reported. I need to collect more results and see the
> > impact to other benchmarks of interest before I have more to share on
> > that.
> 
> Hi Darren,
> if you read Vincent's comments carefully, you will find it is
> pointless for you to
> test the wake_affine patchset as you have already got it. in your
> case, sd_llc_id
> is set to sd_cluster level due to PKG_RESOURCES sharing. So with my new
> patchset for wake_affine, it is completely redundant for your machine
> as it works
> with the assumption cluster-> llc. but for your case, llc=cluster, so
> it works in
> cluster->cluster.

Thanks Barry,

Makes sense as described. I did see degradation in the tests we ran with this
patch applied to 5.17-rc3. I'll have to follow up with you on that when I can
dig into it more. I'd be interested in the specifics of your testing to run
something similar. I think you said you were reporting on tbench?

-- 
Darren Hart
Ampere Computing / OS and Kernel