Received-SPF: pass (google.com: domain of linux-kernel+bounces-71324-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Mon, 19 Feb 2024 20:35:21 +0800
From: Chen Yu <yu.c.chen@intel.com>
To: Abel Wu <wuyun.abel@bytedance.com>
CC: Tiwei Bie <tiwei.btw@antgroup.com>, kernel test robot
	<oliver.sang@intel.com>, <oe-lkp@lists.linux.dev>, <lkp@intel.com>,
	<linux-kernel@vger.kernel.org>, Peter Zijlstra <peterz@infradead.org>,
	<aubrey.li@linux.intel.com>, Honglei Wang <wanghonglei@didichuxing.com>,
	Aaron Lu <aaron.lu@intel.com>
Subject: Re: Re: [linus:master] [sched/eevdf] 2227a957e1:
 BUG:kernel_NULL_pointer_dereference,address
Message-ID: <ZdNLCYnJzPM2Ih2j@chenyu5-mobl2>
References: <202401301012.2ed95df0-oliver.sang@intel.com>
 <23cbb613-c8a2-4f07-b83b-fa3104bef642@bytedance.com>
 <8b21e697-c9b9-49aa-a7ad-e88a5d7f9c92@antgroup.com>
 <5f8f4222-a706-45b1-8eb2-fd4553cc57d8@bytedance.com>
Content-Type: text/plain; charset="iso-8859-1"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <5f8f4222-a706-45b1-8eb2-fd4553cc57d8@bytedance.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?yxdNL2kU1Yur6PBZkD/xKwR+YIzhOiihem/L0oXuMxxJC/MCrQA+vWhHnL?=
 =?iso-8859-1?Q?bROCCYeySQafhpkRC4DSjYhSgIy0CyBENvMw+4XfKHFuRU/8lk98NMEcVq?=
 =?iso-8859-1?Q?3aMi0ztweA1KRze07HxsbfOEMy/3AxodoGAmdfMw1y93JYVdKfuvUKHwTL?=
 =?iso-8859-1?Q?zCuQj5yK65HQ//XvXTssvh58Os1ntpO15KOf1Fx1RgBJ1f+N9ZVrBVXtlU?=
 =?iso-8859-1?Q?E3Sd2SjKTUjpO+IzvG95hTai77+in3Yxa9d1rdwsSVJH0hEBDm4F+pa1LP?=
 =?iso-8859-1?Q?VrRT0biIBbePgNS3hFQnibOqlrjHgauE4u2/+78zJzoYhTB+lX94pmlbd0?=
 =?iso-8859-1?Q?Sug7JPuUb80M38T45lRB74SW7XO2PY/+NhmjxYQzwu16ay53GWn8HjEbVq?=
 =?iso-8859-1?Q?0oKNIsf5wE/rI07ZFucUkkssNZyZL+rkJbnhdyi6LXI++xrx7L8zqq9oQc?=
 =?iso-8859-1?Q?h+HKQMexh9yj9tSEuiooucgGmZ86AWIcxuZ2M6+MQYawmnaGRPfTV2xk0r?=
 =?iso-8859-1?Q?zfUCBW0a42thDO7gqCKOIweL35crf3fHOXO4BhVI7bMgU5SDNAfqsR+Jla?=
 =?iso-8859-1?Q?lin6uj17zpe67ZEPr8aMJHa5OFOpVuU0n3ZeAyNFk1Uqs3Xeg04tGSqQG5?=
 =?iso-8859-1?Q?sxFXKw1eljHe/lMWVFhe05fz2f403q4fnTOYxou/KwLJX0SMT9tAbywnWm?=
 =?iso-8859-1?Q?g1rsqB/dUK9NSQQSeRvhNN86GQDRPqZBNGbuq5MMyaAQ0lIiRg0aAozp5g?=
 =?iso-8859-1?Q?4D+gCzEjprX9mPRK2dyjXgfRXyRjXCeGg9FDdGnUseOIvXytPk9rn6t48/?=
 =?iso-8859-1?Q?hXGaDTx25O/rIxGe5XOGSdFWn4n831VICEC0hy+dDqmNYhNQ2VwnrrvZG5?=
 =?iso-8859-1?Q?p0YeOhdaUqGDTbKw/lVGM3cXVICIrnnrE0R1v43JThH62PbxXmQ91HpGuH?=
 =?iso-8859-1?Q?01yevoWeIhItuTr+/Hb+9sYLlIfsj0jo+7A2HrxeHOakY7C2GuSyvBvzI5?=
 =?iso-8859-1?Q?uici6P7zYH/pXIE/ZJI/uJSn7oEnXHxkfUmEQV8ayRkLkp9XYvDmQNOOAB?=
 =?iso-8859-1?Q?A79hk1+Q5MptHszSbCYNWaAusQKQvKDoAIYeXcNZjAhA23DIF2O60yc7Bz?=
 =?iso-8859-1?Q?2C7p62H8s4wdaDiy/D8Z84r7trIvQ1nzL0497vhZU4NE4EShpdV+B2utNq?=
 =?iso-8859-1?Q?XYPhA97vb+rMRwK2tqL74xMJVuskuh2ytLTcrF1DQqo6TLhyJNpADGxys4?=
 =?iso-8859-1?Q?Ia8aHmvMdl1fEei1nNMdVMN6UFL4ENBVXv68oen6/TVD7hq+Zm01pBCDVg?=
 =?iso-8859-1?Q?UlHY7O7+arFaVDLfZhtpdDd8ERHEeX17Rhe5sNxbNEeXToXugniR74jGJm?=
 =?iso-8859-1?Q?h+LQVMqryFXCHI/ykT01iwfarxU22hE1Vya0wBQ9Zrn8FPfLzztkayOz0o?=
 =?iso-8859-1?Q?24+RKSPHpJNsoZn1Cj5vzu3L2mkaX9enWjG5xzro43WYZQiRtJdmg08BnD?=
 =?iso-8859-1?Q?HPelnz7i7Ie++QPrHiXa4z5uoZdWsdaQZQJUoxnRy1z5cqYW58WUvlOUjD?=
 =?iso-8859-1?Q?OaVL7RR/AP6feznQuzk68EzQjEvacztDcOH9+dPsoOzPS/OE37J/G+3duH?=
 =?iso-8859-1?Q?iJ1tXMjwR9VW989vXWud9EmhjJLi8gDuMS?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a1389e30-9e07-4b67-2d4e-08dc3147440f
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6020.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2024 12:35:33.5195
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: FTJ4BvVhCSx85cPiQQ9tS9+XXCAiMmaJ7vkfyX6gai2woOBGJaq7QKL6J0CNXsmOckuXfHWiyRwg8QNl5Kj3tw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB6851
X-OriginatorOrg: intel.com

On 2024-01-31 at 20:28:19 +0800, Abel Wu wrote:
> On 1/31/24 8:10 PM, Tiwei Bie Wrote:
> > On 1/30/24 6:13 PM, Abel Wu wrote:
> > > On 1/30/24 3:24 PM, kernel test robot Wrote:
> > > > 
> > > > [? 512.079810][ T8305] BUG: kernel NULL pointer dereference, address: 0000002c
> > > > [? 512.080897][ T8305] #PF: supervisor read access in kernel mode
> > > > [? 512.081636][ T8305] #PF: error_code(0x0000) - not-present page
> > > > [? 512.082337][ T8305] *pde = 00000000
> > > > [? 512.082829][ T8305] Oops: 0000 [#1] PREEMPT SMP
> > > > [? 512.083407][ T8305] CPU: 1 PID: 8305 Comm: watchdog Tainted: G??????? W??????? N 6.7.0-rc1-00006-g2227a957e1d5 #1 819e6d1a8b887f5f97adb4aed77d98b15504c836
> > > > [? 512.084986][ T8305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> > > > [ 512.086203][ T8305] EIP: set_next_entity (fair.c:?)
> > > 
> > > There was actually a NULL-test in pick_eevdf() before this commit,
> > > but I removed it by intent as I found it impossible to be NULL after
> > > examining 'all' the cases.
> > > 
> > > Also cc Tiwei who once proposed to add this check back.
> > > https://lore.kernel.org/all/20231208112100.18141-1-tiwei.btw@antgroup.com/
> > 
> > Thanks for cc'ing me. That's the case I worried about and why I thought
> > it might be worthwhile to add the sanity check back. I just sent out a
> > new version of the above patch with updated commit log and error message.
> 
> I assuming the real problem is why it *can* be NULL at first place.
> IMHO the NULL check with a fallback selection doesn't solve this, but
> it indeed avoids kernel panic which is absolutely important.

I tried to reproduce this issue locally but with no luck. It might
be related to lkp's environment so I'm thinking of adding the following
change into lkp's test repo to help narrow down: when the pick_eevdf() finds
an NULL candidate, print the whole rb-tree, including each entity's vruntime,
min_vruntime, deadline, etc. So we can figure out why we failed to find a proper
entity, thoughts?

Example output:

[  325.425017] cfs_rq avg_vruntime:-798130176 avg_load:8192 min_vruntime:29296269335
[  325.425247] current se 0xff11000105858080 is on_rq, deadline:29299081075 min_vruntime:29296537185 vruntime:29297356572, load:1024
[  325.425514] Traverse rb-tree from left to right
[  325.425635]  se 0xff11000101a28080 deadline:29293835648 min_vruntime:29290835648 vruntime:29290835648 eligible
[  325.425876]  se 0xff11000113113000 deadline:29298566205 min_vruntime:29290835648 vruntime:29297726206 non-eligible
[  325.426121]  se 0xff110001036cb000 deadline:29298613981 min_vruntime:29290835648 vruntime:29297094477 non-eligible
[  325.426358]  se 0xff11000118148080 deadline:29298616409 min_vruntime:29297255015 vruntime:29297255015 non-eligible
[  325.426597]  se 0xff11000111523000 deadline:29299066492 min_vruntime:29290835648 vruntime:29296949653 non-eligible
[  325.426836]  se 0xff11000106173000 deadline:29299152426 min_vruntime:29296537185 vruntime:29296537185 non-eligible
[  325.427085]  se 0xff11000105f03000 deadline:29299269335 min_vruntime:29296269335 vruntime:29296269335 eligible
[  325.427322]  se 0xff110001043c0080 deadline:29299707737 min_vruntime:29296707737 vruntime:29296707737 non-eligible
[  325.427558] Traverse rb-tree from topdown
[  325.427648]  left se 0xff110001036cb000 deadline:29298613981 min_vruntime:29290835648 vruntime:29297094477 left half eligible
[  325.427917]  left se 0xff11000113113000 deadline:29298566205 min_vruntime:29290835648 vruntime:29297726206 left half eligible
[  325.428171]  left se 0xff11000101a28080 deadline:29293835648 min_vruntime:29290835648 vruntime:29290835648 left half eligible
[  325.428440]  middle se 0xff11000101a28080 deadline:29293835648 min_vruntime:29290835648 vruntime:29290835648 eligible
[  325.428679] Found best:0xff11000101a28080


diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 533547e3c90a..ddacb3dc91b9 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -856,6 +856,69 @@ struct sched_entity *__pick_first_entity(struct cfs_rq *cfs_rq)
 	return __node_2_se(left);
 }
 
+static void debug_print_eevdf(struct cfs_rq *cfs_rq)
+{
+	struct sched_entity *se, *curr = cfs_rq->curr;
+	struct sched_entity *best = NULL;
+	struct rb_node *node = cfs_rq->tasks_timeline.rb_root.rb_node;
+	struct rb_node *n;
+
+	printk("cfs_rq avg_vruntime:%lld avg_load:%lld min_vruntime:%lld\n",
+		cfs_rq->avg_vruntime, cfs_rq->avg_load, cfs_rq->min_vruntime);
+
+	if (curr && curr->on_rq)
+		printk("current on_rq se 0x%lx, deadline:%lld min_vruntime:%lld vruntime:%lld, load:%ld\n",
+			(unsigned long)curr, curr->deadline, curr->min_vruntime,
+			curr->vruntime, scale_load_down(curr->load.weight));
+
+	printk("Traverse rb-tree from left to right\n");
+	for (n = rb_first(&cfs_rq->tasks_timeline.rb_root); n != NULL; n = rb_next(n)) {
+		bool eli;
+
+		se = __node_2_se(n);
+		eli = entity_eligible(cfs_rq, se);
+		printk(" se 0x%lx deadline:%lld min_vruntime:%lld vruntime:%lld %s\n",
+			(unsigned long)se, se->deadline, se->min_vruntime, se->vruntime,
+			eli ? "eligible" : "non-eligible");
+	}
+	printk("Traverse rb-tree from topdown\n");
+	while (node) {
+		struct rb_node *left = node->rb_left;
+		struct sched_entity *se_left = __node_2_se(left);
+		bool eli = false;
+
+		if (left) {
+			eli = vruntime_eligible(cfs_rq,
+						__node_2_se(left)->min_vruntime);
+
+			printk(" left se 0x%lx deadline:%lld min_vruntime:%lld vruntime:%lld %s\n",
+				(unsigned long)se_left, se_left->deadline, se_left->min_vruntime,
+				se_left->vruntime,
+				eli ? "left half eligible" : "left half non-eligible");
+		}
+
+		if (left && eli) {
+			node = left;
+			continue;
+		}
+
+		se = __node_2_se(node);
+
+		eli = entity_eligible(cfs_rq, se);
+		printk(" middle se 0x%lx deadline:%lld min_vruntime:%lld vruntime:%lld %s\n",
+			(unsigned long)se, se->deadline, se->min_vruntime, se->vruntime,
+			eli ? "eligible" : "non-eligible");
+
+		if (eli) {
+			best = se;
+			break;
+		}
+
+		node = node->rb_right;
+	}
+	printk("Found best:0x%lx\n", (unsigned long)best);
+}
+
 /*
  * Earliest Eligible Virtual Deadline First
  *
@@ -937,6 +1000,9 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq)
 	if (!best || (curr && entity_before(curr, best)))
 		best = curr;
 
+	if (unlikely(!best))
+		debug_print_eevdf(cfs_rq);
+
 	return best;
 }
 
-- 
2.34.1

thanks,
Chenyu