Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 6 Jun 2023 15:58:33 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Peter Zijlstra <peterz@infradead.org>
CC:     Marek Szyprowski <m.szyprowski@samsung.com>,
        <linux-kernel@vger.kernel.org>,
        <linux-tip-commits@vger.kernel.org>, "Tejun Heo" <tj@kernel.org>,
        <x86@kernel.org>
Subject: Re: [tip: sched/core] sched/fair: Multi-LLC select_idle_sibling()
Message-ID: <ZH7nKRDi2LCcpTzk@chenyu5-mobl2.ccr.corp.intel.com>
References: <168553468754.404.2298362895524875073.tip-bot2@tip-bot2>
 <CGME20230605152531eucas1p2a10401ec2180696cc9a5f2e94a67adca@eucas1p2.samsung.com>
 <3051ad44-0ac3-e77b-3178-fac7cac3b3f2@samsung.com>
 <20230605190746.GX83892@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230605190746.GX83892@hirez.programming.kicks-ass.net>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?GB1OFbuBMZIYJE1t5Y4CDg/B6+87CtvdrS9rtyuhOyjcHN80sKB9X5Si2xHN?=
 =?us-ascii?Q?PEsxgZKS2TOSEeRGyl9dzZri9RnVzMpbhG97KgS6fXZxDsmg8+R0iXBetjAr?=
 =?us-ascii?Q?qKaXeUoO4jRn0ztzwTLKe1T6Gk2UJMZz328Y1iucZjWT7GVR9Kiek+7ebMTl?=
 =?us-ascii?Q?n9wLKPwFVd/hzIgwRXWh+bKhUwm6UTV2rxE2uR+n4IVzOdhjezP6tdV/GUgl?=
 =?us-ascii?Q?DEJ2VHaakLfITvmcRSQe9ANctzNAIoGG3xOikuRX9p+j8x4aUc5JUyMaG0ir?=
 =?us-ascii?Q?E0PWfPF0y8Sp++uhtCuqR1jOMWoXK2XA0R4B1qCLHCHdcdrCqAu3GlRjQVxo?=
 =?us-ascii?Q?6NQSxKpDuS24GNPCKONLrbl4nk8YQ8bByYQXMP1DCMKsthkyNZiEIwu2PkYA?=
 =?us-ascii?Q?8/mH1/OaPRJ0/5gkl5VeWM0nlkYBxOaKo0LHZO1lg88z3Fbj1iLtqQ5GJTqZ?=
 =?us-ascii?Q?kNzEffLhjlG/PZcl6QCznPKFOBo/xbV2TGtj3OkxwthEfP7MPJRWmcycn2t9?=
 =?us-ascii?Q?/DE3IdopDKvBCymCNImVUzjiCinr3pxLMCzep3VnFtCFpwuocTpu8VFAufY/?=
 =?us-ascii?Q?j3rmsV4jJhBPPsPWFSabnRs0kg/d3f/a88fXgNYh2Mvf5qbJmgEQ/S3Ul8tz?=
 =?us-ascii?Q?VCqrSuxD5FRdJFSV3nhnE/76lRQXTP2n+d6OBJUnyl1W+ObG8Yz+4okHl7e5?=
 =?us-ascii?Q?UvF89UqkaFupqu9+77Ya6z9puXcz3WJU8d0tanbWwXuYQ2ECGGJeP4PhTvQ7?=
 =?us-ascii?Q?n/d7oG+XNhglDtxqIuS2Q97RYNOR4n+p1buJEDgjk9LlEevkR/q2QcZQYNt9?=
 =?us-ascii?Q?yyT+vEXQvPhOC/XDKbIkSMWAcRdzux0XcHrkA8cT5cuB9ItjRm9P/2c9PG50?=
 =?us-ascii?Q?p/mu9yuGXI9XsG4X9vciB0Pmni8NO7WbxJ0RkPTPT/ikJOzY62y4Tr2TaRy0?=
 =?us-ascii?Q?w5RTkl8pIRuKvZBOx+ep446C2JXvudh4s9t0wLfoOH1AUHWt/xmf3uFTbcg9?=
 =?us-ascii?Q?220LubFkaRX0SsoQFXPZrreqo7dLag7cmaXV12e58dZYyo3+lfTKdkEK+SOX?=
 =?us-ascii?Q?WGdjQ8EQ5HeIV5j0Cvv5dy9fULqe/BoaIlTBuM+rrAh1TJqT48W5lIRpr72u?=
 =?us-ascii?Q?TRaPHPf/unIIeV6Oj5odZSp2t1obK6sj8nVLet3ChQPz9igx8AeB8bF9DTkC?=
 =?us-ascii?Q?2+Z0ebG41Lk7CKDee0U+COEpSq2Jd8XsFanDnSGi7KOpSbMZNkoxOwVf6yeO?=
 =?us-ascii?Q?MfgZ0i3W//9Xq2FHEQgkR0zjXsMl0IAWz5aSQFpWQT5OAGYuOwlnN8aPWFOj?=
 =?us-ascii?Q?IX6S/6RWogVgpVVRQeTN3H0S+ZVA2Zi9XuiDPxgbQazEpW9VsN5lXGJznSCQ?=
 =?us-ascii?Q?1HU3ZO995Dsz7SvXG+cBZrSfBOpD2UZF1uIYsHt7BomEuuGtSrY5tDH5weXd?=
 =?us-ascii?Q?fEEOE3hhwZPHf5Rxe3VFQl3vpWLtBndGdkcpTqec3wOc6v7MsEoJzDe1FKhj?=
 =?us-ascii?Q?oDOx+Slw6tum+u2QnUWsqiJ3L0UpvO4l1XjDx1g4lyaGbh7eW2vDwRwyGInZ?=
 =?us-ascii?Q?rHc8zoLSkQPlY5mGZ370QMZJB6SGOVHoXfxVmDAG?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1d902531-2101-407e-9351-08db6663dda0
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2023 07:58:51.1173
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: cM9VcABLW5N+poVp/wIAlB/+NsdL8n5p4TkrsIHD17Wql0DHBRR+//LGABNUaaXoAknFEjjXFDO4uDD6c0u/iA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB7552
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 2023-06-05 at 21:07:46 +0200, Peter Zijlstra wrote:
> On Mon, Jun 05, 2023 at 05:25:30PM +0200, Marek Szyprowski wrote:
> 
> > nfortunately it causes 
> > regression on my ARM 64bit Exynos5433-based TM2e test board during the 
> > CPU hotplug tests. 
> 
> Can you elucidate an ARM illiterate on the actual topology of that
> machine?
> 
> 
> > CPU: 6 PID: 43 Comm: cpuhp/6 Not tainted 6.4.0-rc1+ #13640
> > Hardware name: Samsung TM2E board (DT)
> > pstate: 000000c5 (nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> > pc : __bitmap_and+0x4c/0x78
> > lr : select_idle_cpu+0x64/0x450
> 
> Btw, where is lr at? Is that perhaps per_cpu(sd_llc) being NULL  or
> something?
> 
> 
> > > ---
> > >   kernel/sched/fair.c     | 38 ++++++++++++++++++++++++++++++++++++++
> > >   kernel/sched/features.h |  1 +
> > >   2 files changed, 39 insertions(+)
> > >
> > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > index 48b6f0c..0172458 100644
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -7028,6 +7028,38 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
> > >   }
> > >   
> > >   /*
> > > + * For the multiple-LLC per node case, make sure to try the other LLC's if the
> > > + * local LLC comes up empty.
> > > + */
> > > +static int
> > > +select_idle_node(struct task_struct *p, struct sched_domain *sd, int target)
> > > +{
> > > +	struct sched_domain *parent = sd->parent;
> > > +	struct sched_group *sg;
> > > +
> > > +	/* Make sure to not cross nodes. */
> > > +	if (!parent || parent->flags & SD_NUMA)
> > > +		return -1;
> > > +
> > > +	sg = parent->groups;
> > > +	do {
> > > +		int cpu = cpumask_first(sched_group_span(sg));
> > > +		struct sched_domain *sd_child;
> > > +
> > > +		sd_child = per_cpu(sd_llc, cpu);
> 
> IOW, sd_child end up NULL?
>
Just wonder if we can use rcu_dereference(per_cpu(sd_llc, cpu)),
and cpu offline/online could modify sd_llc pointer. (We used rcu dereference
for sd_llc in other places)

thanks,
Chenyu