Received-SPF: pass (google.com: domain of linux-kernel+bounces-150478-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <2b9f7617f2b2130bb6270504ec3858f15d463f1d.camel@gmx.de>
Subject: Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
From: Mike Galbraith <efault@gmx.de>
To: K Prateek Nayak <kprateek.nayak@amd.com>, Peter Zijlstra
 <peterz@infradead.org>, mingo@redhat.com, juri.lelli@redhat.com, 
 vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, 
 bsegall@google.com, mgorman@suse.de, bristot@redhat.com,
 vschneid@redhat.com,  linux-kernel@vger.kernel.org
Cc: wuyun.abel@bytedance.com, tglx@linutronix.de, Chen Yu
 <yu.c.chen@intel.com>,  Oliver Sang <oliver.sang@intel.com>
Date: Thu, 18 Apr 2024 18:24:59 +0200
In-Reply-To: <747627a1414f1f33d0c237f555494149d6937800.camel@gmx.de>
References: <20240405102754.435410987@infradead.org>
	 <20240405110010.631664251@infradead.org>
	 <557be85d-e1c1-0835-eebd-f76e32456179@amd.com>
	 <ec6f811b9977eeef1b3f1b3fb951fda066fd95f5.camel@gmx.de>
	 <14330cf4-8d9e-1e55-7717-653b800e5cee@amd.com>
	 <747627a1414f1f33d0c237f555494149d6937800.camel@gmx.de>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.42.4 
Precedence: bulk
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
UI-OutboundReport: notjunk:1;M01:P0:p38Fj4GWluY=;yF7PGIu3lEdCRwJUedh/sSD5EP5
 3Hb7FdJWS3MU9Win8neI+ZvtXhXy+pzA2YY4DK2d+NAPl0scCSEmISHRK83QWBdasyLULUfVx
 7QNJ54i/JkdQHsYxFJVgXSSkZ4fDVQAqLGWtW+mzem/mfkDutir4yj5F5uSbnWyr2VfEdl6EK
 3FHW8DkpjH0t79b6FWAX+8HyK3EdvsSVwyQFP3bp+AOZXQsvRmIPDh/fsTO35nBT80VPCAnGx
 laIUN/LqzxUlFa1cFVnmnggas5X4lyIgmbbNTu89slim6tc5lFonNjVYx0bVyBHXGTJIox6Nb
 Njgo+jsEQLePd/0PVn6rYf9HgCI0FD8/LHDTJ1Nc4ZFac5ieKG/oDoFXwpTedLqzjIstfC+zU
 Oecif4YJX0QxRh1ol2kCvaSKrZLvYyuU5gigth7zcT4h96ys+h44k4/6ZzSXdt9so7fSN4tyg
 9cw8vzQCzcnc1p0KPmjwcF1A1tgxIDEZku9UTm2cS5eizUl26h4pekiuci2vSJO7RrSDj6oZ9
 QDv7cQXnyYgapFvTdQidDRmx4jFjBYP1t8IFty+jw5d6aS+vZa8xPoce0jT/DbvFf4uTS0FVW
 9Uekyo6CBezsU9XEnUTAtw7iHUavSZlJsFq+wI/VnbY9J2fahFY5e/I4lCnHVSx9zu9tJigUn
 hR9KxyXij9QDxR3WqCz65SFJOU+OLVMKKGjpWaaorpWeLy0pIz2+mV1WpsB3e7/hU0IHIs/Gq
 VnUSIAv3RCAcx/LHU65oA99SX8f1PC8CriCcp0rlGLCxcLMncxw/iWBnGOblgOcbMTlRBXQ/l
 OlE88K4KNWobow+VBs8CzkRbJo7AdTfyMiFitZv0z9isQ=

Greetings,

I tossed a couple rocks at it today, and seem to have hit the little
bugger.  The root cause seems to be doing the delay dequeue business on
exiting tasks.  Hunk #1 of hacklet below seems to quell the explosions.

crash> bt -sx
PID: 21722    TASK: ffff88815710ee40  CPU: 6    COMMAND: "hackbench"
 #0 [ffff88822773bb90] machine_kexec+0x156 at ffffffff810642d6
 #1 [ffff88822773bbe0] __crash_kexec+0xd7 at ffffffff81152a07
 #2 [ffff88822773bc98] crash_kexec+0x23 at ffffffff81152eb3
 #3 [ffff88822773bca8] oops_end+0xbe at ffffffff810329be
 #4 [ffff88822773bcc8] page_fault_oops+0x81 at ffffffff81071951
 #5 [ffff88822773bd28] exc_page_fault+0x62 at ffffffff8194e9e2
 #6 [ffff88822773bd50] asm_exc_page_fault+0x22 at ffffffff81a00ba2
    [exception RIP: pick_next_task_fair+178]
    RIP: ffffffff810d8d12  RSP: ffff88822773be00  RFLAGS: 00010006
    RAX: ffff88813cb780b8  RBX: ffff88840edb0e80  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: ffff88813cb78080  RDI: ffff88840ec30f00
    RBP: ffff88813cb78000   R8: ffff88815710eec0   R9: 0000000000000001
    R10: ffff88822773bdc8  R11: 0000000000000013  R12: 0000000000030e80
    R13: ffff88815710ee40  R14: ffff88813cb78080  R15: ffff88815710ee40
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffff88822773be28] __schedule+0x10d at ffffffff8195865d
 #8 [ffff88822773be98] do_task_dead+0x3e at ffffffff810cc00e
 #9 [ffff88822773beb0] do_exit+0x770 at ffffffff8108f0e0
#10 [ffff88822773bf00] do_group_exit+0x2c at ffffffff8108f64c
#11 [ffff88822773bf28] __x64_sys_exit_group+0x14 at ffffffff8108f6f4
#12 [ffff88822773bf30] do_syscall_64+0x57 at ffffffff8194ac37
#13 [ffff88822773bf50] entry_SYSCALL_64_after_hwframe+0x76 at ffffffff81a0=
012b
    RIP: 00007f4f2aa76136  RSP: 00007ffcbba84748  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 0000000000000000  RCX: 00007f4f2aa76136
    RDX: 0000000000000000  RSI: 000000000000003c  RDI: 0000000000000000
    RBP: 00007f4f2ab86970   R8: 00000000000000e7   R9: ffffffffffffff80
    R10: 0000000000000004  R11: 0000000000000246  R12: 00007f4f2ab86970
    R13: 0000000000000001  R14: 00007f4f2ab8a328  R15: 0000000000000000
    ORIG_RAX: 00000000000000e7  CS: 0033  SS: 002b
crash> task_struct ffff88815710ee40 | grep sched_delayed
    sched_delayed =3D 1 '\001',
crash>

=2D--
 kernel/sched/fair.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

=2D-- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5374,6 +5374,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
 		update_curr(cfs_rq);

 		if (sched_feat(DELAY_DEQUEUE) && sleep &&
+		    !(entity_is_task(se) && (task_of(se)->flags & PF_EXITING)) &&
 		    !entity_eligible(cfs_rq, se)) {
 			if (cfs_rq->next =3D=3D se)
 				cfs_rq->next =3D NULL;
@@ -5495,14 +5496,14 @@ pick_next_entity(struct rq *rq, struct c
 	}

 	struct sched_entity *se =3D pick_eevdf(cfs_rq);
-	if (se->sched_delayed) {
+	while (se && se->sched_delayed) {
 		dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
 		SCHED_WARN_ON(se->sched_delayed);
 		SCHED_WARN_ON(se->on_rq);
 		if (sched_feat(DELAY_ZERO) && se->vlag > 0)
 			se->vlag =3D 0;

-		return NULL;
+		se =3D pick_eevdf(cfs_rq);
 	}
 	return se;
 }