Received-SPF: pass (google.com: domain of linux-kernel+bounces-191571-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
From: Ankur Arora <ankur.a.arora@oracle.com>
To: linux-kernel@vger.kernel.org
Cc: tglx@linutronix.de, peterz@infradead.org, torvalds@linux-foundation.org,
        paulmck@kernel.org, rostedt@goodmis.org, mark.rutland@arm.com,
        juri.lelli@redhat.com, joel@joelfernandes.org, raghavendra.kt@amd.com,
        sshegde@linux.ibm.com, boris.ostrovsky@oracle.com,
        konrad.wilk@oracle.com, Ankur Arora <ankur.a.arora@oracle.com>,
        Ingo Molnar <mingo@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>
Subject: [PATCH v2 26/35] sched/fair: handle tick expiry under lazy preemption
Date: Mon, 27 May 2024 17:35:12 -0700
Message-Id: <20240528003521.979836-27-ankur.a.arora@oracle.com>
In-Reply-To: <20240528003521.979836-1-ankur.a.arora@oracle.com>
References: <20240528003521.979836-1-ankur.a.arora@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?Sfs/R3Qr/hcY9PR5whTQHGx4B3/01PnIjnMVXG6/WH1C8ostsKS5xoWGnvPh?=
 =?us-ascii?Q?oTpU+ghSwLasISBBMj6a2mWAIW9Ym4koUBghWQOtkB4CWSzVxIfadPKxbfjm?=
 =?us-ascii?Q?tt9zEmMp7/7OAAxfxEMdhIe8639RvAwm0mSXICuEDsxh2MmMbjnarBhchf/B?=
 =?us-ascii?Q?tf8x0LScpo83QwdKpe9LhUVfbpTM+/6wR2SKrThpsnFXAQ9QDkDsZG2faN1t?=
 =?us-ascii?Q?YvZb8w7u+qI8GOPeNYMj8wMBTLjESk95dqtEIN175r8m3D8FfOfBSR03RKa1?=
 =?us-ascii?Q?GGn46peOK3e2vuTwAdJD+Xl7A4IAAgFoLmzUTVdXRjxZQbng2jjYZIvZFU8A?=
 =?us-ascii?Q?InRkcOgYiDl0L+PGMeTO13g9ndyUzMn2+ur7BB1uFDnYnl1ikOP3Doi2mQ/E?=
 =?us-ascii?Q?1XvL+7CarGhaiQDIKBJsQCqtl2G5bT+7qRqhO03YfsNHW/eZwWQK1pqW/tWC?=
 =?us-ascii?Q?bxIzrQfhGHjxaoTwU871+ogVqXdZhQPP0LWbybhmGixPGWivud6RBhRu1zI9?=
 =?us-ascii?Q?NRV6D8kMuhD7ziM0rXIDljkAhd9eTMkKzlX1UsLyHyVCzAHgmS0leLkr2kgZ?=
 =?us-ascii?Q?goL+UknZ3msZ+0rcb8O03WeutvYlvcsHa7achM26dJ12SyeH3o/aq/u2FM3M?=
 =?us-ascii?Q?vDcruZFZzKYhJIjkdWq4/Ryfgm6SKjba/gq4IhU4wwOFTX9vpktadKbnyHXY?=
 =?us-ascii?Q?58D6+4nRUK67J+Rd4iRr8eZclyOOG/ntTXMstxVOT+VIRcjGy4nzi1o7FvaE?=
 =?us-ascii?Q?n4Ll/znz+WJxV/buIt8y92WMEl5SCxHNTJcHujSJ/am2o50fkA5/RRYRQvRc?=
 =?us-ascii?Q?kQ4vHGs8TuFE43f1ZpMv5zGz2TzuDSxD5bMOFCOHBA/XPqn3vXvew7TAUHjJ?=
 =?us-ascii?Q?5jtmeKo5/mSqrVyxC/hSMg2ivX20dyyyJUUQ4zXvS0jZQpDwJVzjIICyqUHk?=
 =?us-ascii?Q?sB61vuMBHBikud8H9lChM7xW4bqHcrP05CurPZltKdibSxQdstOl/uGHl9oQ?=
 =?us-ascii?Q?hWm3EqK24AvCfLwsEusNQGGVcKCrlxa1CdSa5X+rRw0fTUf1mz97bW9woBBJ?=
 =?us-ascii?Q?r6O/Nw2+CTmB6CrrA/CGwY2SQBkIimHG9B3MSvioQbEvZ04NUc+TzpfsaLY/?=
 =?us-ascii?Q?B0PE5zPjSUNZOFotDngtVumjpGVBwyEG8VJkr+piNlCV9sAwJGVfAtZs3rp7?=
 =?us-ascii?Q?E6fw3aG8fqUv6y8gvXNj0qu4LriJIaoPpBQzqlBDHetBXA9SFeB02Wmo8kjJ?=
 =?us-ascii?Q?M0XgGKNlxh6ZyiB4fCZFN6y8FYt61J/ZLbLATxeqGMH+eJicQ9o+uKJZSYNZ?=
 =?us-ascii?Q?ufmsczsYJ036WpBRUds0xxC6e1gqas2uG5fHeHoIfguBfafBU1Kyum0GhSnP?=
 =?us-ascii?Q?4GM9aiE5oXPt0J/WYMWF2UDQOsWuWZZ712YyH/V9E8vwPTcKZO/3eNPlt0Ov?=
 =?us-ascii?Q?bLFqWGALFkJbMwvY2KYUUe2hpCJ1TTMFNt17sez9vw+bDV4ZMt9UoPPQVYKm?=
 =?us-ascii?Q?bpYt1XS38Vh0if83AaUm6bUrdwxX2SGR+n4D2GYqcSOHS/kef0fh7eKl4T4x?=
 =?us-ascii?Q?zlCfauFPc+R+vrdtN8QuhPm4dLSc/K+UPSfVYcSSOX1aJCvS/7xR5H3hwcFO?=
 =?us-ascii?Q?rw=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	K7vNHAT0neQdREjn5oqEEQfrCKVruiJWqpl0yRmlJ/X5l88LZUX9uO3ifXjDkbPFi+lIAE8xwlo1ut2KLu8tD2nnvz3mEQVwkKHENq64wQkUSwHrm7UpPoGh1ZZ5/qq21B0C+GlGhUxQBDUfxcJSJi48SZg5iMWiB8Ayu4dlZ+Y+JZmFbT3zmq6t8UMAwn6X99BOEigjeLgUYKdlBle5i8HdrjVF5mMF1UFzdqfZKEZcY0W+mKdFIs7b2UVnBsdgdEz+mFS7Um/hsxCaNsylMH5hfsY4cXpmcyXQONgCUmlAl2LSLDTa1dYnhuobE+vDsvAWzYhCpLiyIRDfe2RYKzbJRYbLMwrS0Y3AlQiLb7G3BFsGeCLzmqm/oqW6nMNU/w++4HDQ7oEKekqS6YoMRN929g2464Ouk8qt9NAIrWP2ndsZlPyPFg94ULCN2Cjasn8SSzS2ZGZg5ezpg7EA2JhdNsELslR70QTph1/JVyoC6/CoKkUVWtiBCR4laSGYj77uLEHaR7PYOkwFLkFlbgu0A8PnezTwXF2hh+gvTWiG4+V2cTBZcgS7q9QelR03S/8g+fOw3emlVOUjqaCQyhL72yjcubEfC5gDE+zqyvY=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8d6ff32f-eba1-453d-d3f1-08dc7eae3f2d
X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2024 00:36:43.0166
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: sjGLaMyfYGrQ14ieI6ljkyHuod1XUGl2pdytUaIGDgLXCPVD+OzTiNp1sPoAXdeDhQkP1GsD+Msy1PTj3c9iU/7gg2Wwr/+AxJgzVvWBOvk=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB7177
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16
 definitions=2024-05-27_06,2024-05-27_01,2024-05-17_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 mlxscore=0
 mlxlogscore=999 phishscore=0 spamscore=0 adultscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000
 definitions=main-2405280003
X-Proofpoint-GUID: 9mdEMb6iCce4B2VL2XVt_-MHF0djy8XD
X-Proofpoint-ORIG-GUID: 9mdEMb6iCce4B2VL2XVt_-MHF0djy8XD

The default policy for lazy scheduling is to schedule in exit-to-user.
So, we do that for all but deadline tasks. For deadline tasks once a
task is not leftmost, force it to be scheduled away.

Always scheduling lazily, however, runs into the 'hog' problem -- the
target task might be running in the kernel and might not relinquish
CPU on its own.

Handle that by upgrading the ignored tif_resched(RESCHED_LAZY) bit to
tif_resched(RESCHED_NOW) at the next tick.

Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Originally-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/lkml/87jzshhexi.ffs@tglx/
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 kernel/sched/core.c     | 8 ++++++++
 kernel/sched/deadline.c | 5 ++++-
 kernel/sched/fair.c     | 2 +-
 kernel/sched/rt.c       | 2 +-
 kernel/sched/sched.h    | 6 ++++++
 5 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index e838328d93d1..2bc7f636267d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1051,6 +1051,14 @@ static resched_t resched_opt_translate(struct task_struct *curr,
 	if (is_idle_task(curr))
 		return RESCHED_NOW;
 
+	if (opt == RESCHED_TICK &&
+	    unlikely(__test_tsk_need_resched(curr, RESCHED_LAZY)))
+		/*
+		 * If the task hasn't switched away by the second tick,
+		 * force it away by upgrading to TIF_NEED_RESCHED.
+		 */
+		return RESCHED_NOW;
+
 	return RESCHED_LAZY;
 }
 
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index d24d6bfee293..cb0dd77508b1 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -1378,8 +1378,11 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64
 				enqueue_task_dl(rq, dl_task_of(dl_se), ENQUEUE_REPLENISH);
 		}
 
+		/*
+		 * We are not leftmost anymore. Reschedule straight away.
+		 */
 		if (!is_leftmost(dl_se, &rq->dl))
-			resched_curr(rq);
+			__resched_curr(rq, RESCHED_FORCE);
 	}
 
 	/*
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index dd34709f294c..faa6afe0af0d 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -12615,7 +12615,7 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 	}
 
 	if (resched)
-		resched_curr(rq);
+		resched_curr_tick(rq);
 
 	if (static_branch_unlikely(&sched_numa_balancing))
 		task_tick_numa(rq, curr);
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index f0a6c9bb890b..4713783bbdef 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -1023,7 +1023,7 @@ static void update_curr_rt(struct rq *rq)
 			rt_rq->rt_time += delta_exec;
 			exceeded = sched_rt_runtime_exceeded(rt_rq);
 			if (exceeded)
-				resched_curr(rq);
+				resched_curr_tick(rq);
 			raw_spin_unlock(&rt_rq->rt_runtime_lock);
 			if (exceeded)
 				do_start_rt_bandwidth(sched_rt_bandwidth(rt_rq));
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index e5e4747fbef2..107c5fc2b7bb 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2467,6 +2467,7 @@ extern void reweight_task(struct task_struct *p, int prio);
 enum resched_opt {
 	RESCHED_DEFAULT,
 	RESCHED_FORCE,
+	RESCHED_TICK,
 };
 
 extern void __resched_curr(struct rq *rq, enum resched_opt opt);
@@ -2476,6 +2477,11 @@ static inline void resched_curr(struct rq *rq)
 	__resched_curr(rq, RESCHED_DEFAULT);
 }
 
+static inline void resched_curr_tick(struct rq *rq)
+{
+	__resched_curr(rq, RESCHED_TICK);
+}
+
 extern void resched_cpu(int cpu);
 
 extern struct rt_bandwidth def_rt_bandwidth;
-- 
2.31.1