Received-SPF: pass (google.com: domain of linux-kernel+bounces-144721-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Mon, 15 Apr 2024 16:03:01 +0800
From: Chen Yu <yu.c.chen@intel.com>
To: Peter Zijlstra <peterz@infradead.org>
CC: Abel Wu <wuyun.abel@bytedance.com>, Ingo Molnar <mingo@redhat.com>,
	Vincent Guittot <vincent.guittot@linaro.org>, Juri Lelli
	<juri.lelli@redhat.com>, Tim Chen <tim.c.chen@intel.com>, Tiwei Bie
	<tiwei.btw@antgroup.com>, Honglei Wang <wanghonglei@didichuxing.com>, "Aaron
 Lu" <aaron.lu@intel.com>, Chen Yu <yu.chen.surf@gmail.com>, Breno Leitao
	<leitao@debian.org>, <linux-kernel@vger.kernel.org>, kernel test robot
	<oliver.sang@intel.com>
Subject: Re: [RFC PATCH] sched/eevdf: Return leftmost entity in pick_eevdf()
 if no eligible entity is found
Message-ID: <ZhzfNU0rsscFnTm4@chenyu5-mobl2>
References: <20240226082349.302363-1-yu.c.chen@intel.com>
 <758ebf4e-ee84-414b-99ec-182537bcc168@bytedance.com>
 <20240408115833.GF21904@noisy.programming.kicks-ass.net>
 <ZhPtCyRmPxa0DpMe@chenyu5-mobl2>
 <20240409092104.GA2665@noisy.programming.kicks-ass.net>
 <20240415072251.GA12673@noisy.programming.kicks-ass.net>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240415072251.GA12673@noisy.programming.kicks-ass.net>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Gr6gTEwZ4ACTZVGJKLQ+MeBS0sxKpZD4kDz47/p21aKcB1371kDx78Mj7d8E?=
 =?us-ascii?Q?HlnoI2+aA1CoXI23tXtCh6D7/TTvXnU5qRyivl1zqj/asPnNnW9bzzEaXHxS?=
 =?us-ascii?Q?mzvs8dtdnObFDEQKrFwjTvNyIs0VMKuWz6qkSRiNSBhWq3vzGjEPVm86Llx/?=
 =?us-ascii?Q?vYqCleWZZWbG0w8EoR3ABgcvHJxepGx+vERVbZsW0topx5kaw3NYG6U7fjPW?=
 =?us-ascii?Q?0b6AFdTpQ7BH3WZRO+DDXK/L4eaHzlKuGNZhOZSWV17GPRrMqa94JVY5MYLV?=
 =?us-ascii?Q?+cXnDP6OHgCjccqK9haZTjKpIAB8Sgmr5tlwc7uFwH4EgpcBqOMRYfXP9hUa?=
 =?us-ascii?Q?FNKUsfCb85UDXhzoloddcpEX+5fnmeX6fodKj7JMCMEk8E7b6QtDR0D/7J1K?=
 =?us-ascii?Q?pevs82FGUGHB65+ZXbPgF1hSB3E/xjKdhopqnVrQBPqFKVonrVPAGWN2eLAs?=
 =?us-ascii?Q?lWXCBDd5kOe4UtkkvTJqoukPi9yOHTl/mU6x4k9cWYT6NYRm2abVjW2lxRBI?=
 =?us-ascii?Q?5YzM98Gi703VAFnqe2tfiJDq8mec+hm4YbTvlingOW4KSlRl+4JNTtzdto3O?=
 =?us-ascii?Q?HNgp3wPQSL1kGLvmhOq4/tfzJtyGanPDHcZAyRdwJ+BXchxMmOOMuL5N+etA?=
 =?us-ascii?Q?gqoHTwnMUceehlrV78dUO3rqcNQzZ2tW3mCkBuBsn38oQKFmJfh8GzdAdI75?=
 =?us-ascii?Q?nHaZLgKHrUq84OqPoV5FQ2vVPpe0IfgWVZ1Zpw64eaqA/EuiNmiOUVxVivpa?=
 =?us-ascii?Q?J++4WrGfpGCdCta7j6xPWJGuqATnzYHpb5c1zCzk2NlJu335Gt0kB0xi/YHc?=
 =?us-ascii?Q?0cXhLQOJlHRHZwphDElNwgqvUbiWLZSaqMv9ysHohWstjvCP6e34oTRxWXp+?=
 =?us-ascii?Q?n0I34KTOBzmtTdfHG+Y9Tn2P/6pJ5c+Zz10Vat5HYNVfmk2V7j8H5qabYaPZ?=
 =?us-ascii?Q?4rgtlaAmlXmA4+ijBoDeUI2VMcM1k/Renb22eyij7TrgAbACv2e/rMzk+EOo?=
 =?us-ascii?Q?4ZfGh9ezlWuEdXLH1VEnbnycr5jiaYqhXVBZtajKTThOUiFEidA4XvUqAjyL?=
 =?us-ascii?Q?w6kbYx/M8ckAzPa+Hx03sDwWZOYszX0wUlLGjfXktsArhOpSVznGFaacEMr9?=
 =?us-ascii?Q?hR2uvg/AiFZ8zfK7SBIMr03tcmzYkr9QBvVESJtXzlkQ9JY7MxOi68025471?=
 =?us-ascii?Q?0a70pC9FRak/F0dHKlO1z9hubv3vCEYeaO6h7L/AhmVKUfJXJDB8WTp5e1NY?=
 =?us-ascii?Q?xkeqxpJqABWBpTVNpf7aYli1pZgYHqf7Otr1V+MWfznoT9oUQ/XO6sV6fRCG?=
 =?us-ascii?Q?ctHEgbiQwRl8yR6/1LSvSO1vngSpK/wavG1sBPT7912vNHgrEI8Ng89hOm7R?=
 =?us-ascii?Q?w7FLY0PbLYJvbxqAZ6eXcjh36DuaRcehuUwZHFCqssTOAYjNeKO1WOgCHGgQ?=
 =?us-ascii?Q?WmurvY1+WUfoofX4lvoEw/a821JB/9YokWUPS7/Kzn9YM3XLTotYEIFNfkq9?=
 =?us-ascii?Q?GlIH3AZm488GNt3PYKnRoDrv46QYMxDfBD7KC+n85VzIopV13YnN5IklINcw?=
 =?us-ascii?Q?NRwW/HIsEkEIgvoOUD/JnLnjbyqsCDvdFz94NnTv?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c29e97f3-0a71-4293-2d50-08dc5d2281b5
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6020.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Apr 2024 08:03:16.7637
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 1Cr0dsSgjn/YqeJThWTU38WUiIRALrWzat1FHZUJg0fpsOH8xsJMwqWTKTowcvvLvPaIeY0hA0fOnT/J3xB5JA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB5766
X-OriginatorOrg: intel.com

On 2024-04-15 at 09:22:51 +0200, Peter Zijlstra wrote:
> On Tue, Apr 09, 2024 at 11:21:04AM +0200, Peter Zijlstra wrote:
> 
> > Is there any sane way to reproduce this, and how often does it happen?
> 
> This, how do you all make it go bang?

It was reproduced in lkp's environment, and originally reported here:
https://lore.kernel.org/lkml/202401301012.2ed95df0-oliver.sang@intel.com/

It is a trinity test on a vm guest, and seems like be triggered after some futex
test.  And it was reproduced at a rate of 23/999 according to that report.
Previously I could not reproduce it locally, so lkp has helped test my debug
patch in their environment and got the clue that it was broken by s64 overflow.

Breno told me that he has reproduce this issue with KASAN on and using:
'stress-ng -a 20', but I can not reproduce it locally neither.

I'm thinking of creating a debug patch to trace all the changes related to
cfs_rq->avg_vruntime in avg_vruntime_add()/sub().
To see how cfs_rq->avg_vruntime gets far behind the cfs_rq->min_vruntime, which
caused the overflow. My understanding is that, the se's vruntime is got from
cfs_rq->avg_vruntime in place_entity(), if the se's vruntime gets an extrem
smaller value than cfs_rq->min_vruntime, then it might indicate that there could
be something wrong with the update of cfs_rq->avg_vruntime.
Then lkp could help us to further debug.

Or do you have any suggestion/suspect that how to narrow down this, I could
try as you suggest.

thanks,
Chenyu