Received-SPF: pass (google.com: domain of linux-kernel+bounces-191570-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: Ankur Arora <ankur.a.arora@oracle.com>
To: linux-kernel@vger.kernel.org
Cc: tglx@linutronix.de, peterz@infradead.org, torvalds@linux-foundation.org,
        paulmck@kernel.org, rostedt@goodmis.org, mark.rutland@arm.com,
        juri.lelli@redhat.com, joel@joelfernandes.org, raghavendra.kt@amd.com,
        sshegde@linux.ibm.com, boris.ostrovsky@oracle.com,
        konrad.wilk@oracle.com, Ankur Arora <ankur.a.arora@oracle.com>,
        Ingo Molnar <mingo@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>
Subject: [PATCH v2 25/35] sched/fair: refactor update_curr(), entity_tick()
Date: Mon, 27 May 2024 17:35:11 -0700
Message-Id: <20240528003521.979836-26-ankur.a.arora@oracle.com>
In-Reply-To: <20240528003521.979836-1-ankur.a.arora@oracle.com>
References: <20240528003521.979836-1-ankur.a.arora@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?bUbi0S3peedFKMtk2SQkLcMPv+NuItkicJg2hJU39gWhXaO70uKvBA3xovgf?=
 =?us-ascii?Q?6F3DrspFmr351CnbDCtIP7bwGIIFr0M1kfM2qy9ijv+0kkB2HqBLy1cL7cqs?=
 =?us-ascii?Q?kVto3BrUw3+YU0l4byDwKEne2c3bDDljyNH87g2Fo3m9hFSoRFDNCGDmOR7z?=
 =?us-ascii?Q?4WsH9gHxHzpiepDsGQi5TNXJYgwSKbQhXSIZr08cl3NBx095WAdvpS1zfo3Z?=
 =?us-ascii?Q?dXkxxgOG60KUE4tTWmOoN41SekkVtzDzQq1nSWHqcwW8tKA/PDq5BIK2jXD3?=
 =?us-ascii?Q?sUlTzcewXdyqQL+8sgohjO7B17+HvxuHlZs1F4PYtpfPN1/jw5HkSTIPMppT?=
 =?us-ascii?Q?ABSiu7TyJSJbfXU4n5+NjuwI2kRDrAJ86fc+2OlYbsFUeqvh9th9TEe/Ks0d?=
 =?us-ascii?Q?8cF3ln2r9W3Df95JHDwxUt+niVOTA9lKA0VMWO2z2jStsXU22nS5T5rv5jlf?=
 =?us-ascii?Q?a1a7G5QJRs3yc9UtR5PVvuNWQrbif6KnI+jB9x2nT/hwcyGrEOZaB6cCcv9U?=
 =?us-ascii?Q?r/T9dCCbEwHlMyx4d3xJz8unzOfpx39rSbgzReecaD2Awpgcp1F405HOD3Hs?=
 =?us-ascii?Q?d5CnJPTAvLcPWhQgXysnKFxA0kZnQaQn100bGD/DdUxPS2QeJAhbe5zNQIhR?=
 =?us-ascii?Q?0jpq87GjxiWLY3kA69/MwPGTwqRSDl8oYxMwt+Dm/Ti7gW4Z/26Sy3cjLP49?=
 =?us-ascii?Q?s/FN+9NPa4UiC9GQ0T9/mItFtPfDPjhpoYpJ5fh89LPHCG2eNNJAR/RZb4ii?=
 =?us-ascii?Q?/padnLQvL3W32l7nawq0OhTZEXIKlUt5KrmVOCNUjilKLw4+BT6mvXbUPh+x?=
 =?us-ascii?Q?8pvBA0ALq6dAM6miLE0ZzkShWjK1xb+z8b3SWxmCgRBXoh/iiVFYBay0ZNR1?=
 =?us-ascii?Q?2hZM5LfA/6ZM7h4HqkkxwYT0Fz4NaQ+UCAuLWU8QffVlHFXG/jSWd8hLvZJb?=
 =?us-ascii?Q?HaYQChu/XvAlsjDonX42hCpWs4XOGOfzA4+Xvz3xMXQ1p2quEOPshmb/li70?=
 =?us-ascii?Q?gIXCSGLRaxo5zTy+9c25tC9lEX4DFlFHSiYzbSVlsMyQ+yxMPt0c/jNc1p0t?=
 =?us-ascii?Q?bcEHP6qyXZpffwaul6ByHbWNP5BeV957scVU4YVVLZ638siRTS91zKPZCyIt?=
 =?us-ascii?Q?uJPjd8wnPy02S7keCKiLt7K+arBvcjf8xOULVSaupDu6A5J19CwwSKK8XI22?=
 =?us-ascii?Q?DySHRxmr429nJ1xB1fVDgG6s/oHN2F0xwI95DTn7OdVaUmoXgZAZGSjT5CKV?=
 =?us-ascii?Q?X2MvHR30+67k715xG4qELTYC4dSH5d+X2xvy+bYpxIHIGsDTil0oXJqS0fmo?=
 =?us-ascii?Q?/I+0X4bhzuzZWk5OInrLupYYPhzj3t6HooppcjsD01KzFAJo+SHgVNTJ4OhT?=
 =?us-ascii?Q?caSQmZjcNJ3c8RCnI1LbhaDP14oJK7kr74DIaIit5QC6fz2LPWvwOufJ3r78?=
 =?us-ascii?Q?vC2dqpXxBE5/uTxbMf4hqOZ6vNPwti/Zu/XzSeXTtvbpqbgsN9UL+iCcV3NG?=
 =?us-ascii?Q?3wzJDsgN5ASa2pGBcUV+SiArxddQcLOS6+nHTdtuYu/mQNQRde7YtcIR3cLK?=
 =?us-ascii?Q?F3ibYOCn/vnDBzhtjNrlZ5fo+oOWI8vFtaxW/MyUPzaQiP+8SG1fxrXXhwZe?=
 =?us-ascii?Q?Eg=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	S2x8mheloa0r3E0qodfFgStPIHaxptYqjVMXE0muiYrWPYQ7Hk+znJu4LDDp0+QvRPt2Pn1Lc8Xge0W8zirJAFxSoKS0FrR+ebgfAVBcE9JyecJVkvvrESxz6xdzf+rK4Cs528lnQys4ME1hZMOjATYwo1C9z5rIvckpUW083sijNmqBW9e8uIVDqu84rxFBf73nMxxgIOwi8R+q3ET6yI/10OGI/n/4vliI+W9fvoGKQqC2nIFBSUNoAqwU32PxB4vpeiBVeSGS/AAXFbCwJJvQsmR7tjEA1eA0ifPyekTefAtZ6b2vKI9p6KEACpsM6f41sf374clqe+Gwnfkb5TKsekUzxUXGLPNp5WLCOL3JhpLrCX1aMsfrXq0El0M+JAfJk/3SIURNfAeoDf20nV2MEzuAML/g3IaXbdfLhtiShodoJNGqERoogSuztuo3FrFAy/HQ+fDDRF0pNEsxjW8/Zw6NLWTq+j4CPTC5RHwvFJmYnSvULPmFHE0HiT+t9V3JHGX4Ti+XCX7WHwQ8VWPZ5hKirBiF5W26WXb8+FUbBcXxqJUxGxozEvy4LwmulOACVuxzwN6zwSIcCCAemmyy9Xmg4t0Lj0d1SXHcXWg=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c0fbf81a-861d-424c-0551-08dc7eae3db8
X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2024 00:36:40.6160
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Prq78Ko0cyviBoRkbF0vdyMKx5QsY1uSw8YoLT1PujeFvJY2OTnyA1mOrwjlNh81Anxi061JS+ZXk596Xq2XMXkblB5cFd0omCSQvZEQGV8=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB7177
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16
 definitions=2024-05-27_06,2024-05-27_01,2024-05-17_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 suspectscore=0
 adultscore=0 phishscore=0 malwarescore=0 mlxlogscore=999 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000
 definitions=main-2405280003
X-Proofpoint-GUID: 5pmUGxdK3pO5T28L4qHVXxjGgt3BqGry
X-Proofpoint-ORIG-GUID: 5pmUGxdK3pO5T28L4qHVXxjGgt3BqGry

When updating the task's runtime statistics via update_curr()
or entity_tick(), we call resched_curr() to resched if needed.

Refactor update_curr() and entity_tick() to only update the stats
and deferring any rescheduling needed to task_tick_fair() or
update_curr().

Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Suggested-by: Peter Ziljstra <peterz@infradead.org>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 kernel/sched/fair.c | 54 ++++++++++++++++++++++-----------------------
 1 file changed, 27 insertions(+), 27 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index c5171c247466..dd34709f294c 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -981,10 +981,10 @@ static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se);
  * XXX: strictly: vd_i += N*r_i/w_i such that: vd_i > ve_i
  * this is probably good enough.
  */
-static void update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
+static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
 	if ((s64)(se->vruntime - se->deadline) < 0)
-		return;
+		return false;
 
 	/*
 	 * For EEVDF the virtual time slope is determined by w_i (iow.
@@ -1002,9 +1002,11 @@ static void update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
 	 * The task has consumed its request, reschedule.
 	 */
 	if (cfs_rq->nr_running > 1) {
-		resched_curr(rq_of(cfs_rq));
 		clear_buddies(cfs_rq, se);
+		return true;
 	}
+
+	return false;
 }
 
 #include "pelt.h"
@@ -1159,26 +1161,35 @@ s64 update_curr_common(struct rq *rq)
 /*
  * Update the current task's runtime statistics.
  */
-static void update_curr(struct cfs_rq *cfs_rq)
+static bool __update_curr(struct cfs_rq *cfs_rq)
 {
 	struct sched_entity *curr = cfs_rq->curr;
 	s64 delta_exec;
+	bool resched;
 
 	if (unlikely(!curr))
-		return;
+		return false;
 
 	delta_exec = update_curr_se(rq_of(cfs_rq), curr);
 	if (unlikely(delta_exec <= 0))
-		return;
+		return false;
 
 	curr->vruntime += calc_delta_fair(delta_exec, curr);
-	update_deadline(cfs_rq, curr);
+	resched = update_deadline(cfs_rq, curr);
 	update_min_vruntime(cfs_rq);
 
 	if (entity_is_task(curr))
 		update_curr_task(task_of(curr), delta_exec);
 
 	account_cfs_rq_runtime(cfs_rq, delta_exec);
+
+	return resched;
+}
+
+static void update_curr(struct cfs_rq *cfs_rq)
+{
+	if (__update_curr(cfs_rq))
+		resched_curr(rq_of(cfs_rq));
 }
 
 static void update_curr_fair(struct rq *rq)
@@ -5499,13 +5510,13 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
 	cfs_rq->curr = NULL;
 }
 
-static void
-entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
+static bool
+entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
 {
 	/*
 	 * Update run-time statistics of the 'current'.
 	 */
-	update_curr(cfs_rq);
+	bool resched = __update_curr(cfs_rq);
 
 	/*
 	 * Ensure that runnable average is periodically updated.
@@ -5513,22 +5524,7 @@ entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
 	update_load_avg(cfs_rq, curr, UPDATE_TG);
 	update_cfs_group(curr);
 
-#ifdef CONFIG_SCHED_HRTICK
-	/*
-	 * queued ticks are scheduled to match the slice, so don't bother
-	 * validating it and just reschedule.
-	 */
-	if (queued) {
-		resched_curr(rq_of(cfs_rq));
-		return;
-	}
-	/*
-	 * don't let the period tick interfere with the hrtick preemption
-	 */
-	if (!sched_feat(DOUBLE_TICK) &&
-			hrtimer_active(&rq_of(cfs_rq)->hrtick_timer))
-		return;
-#endif
+	return resched;
 }
 
 
@@ -12611,12 +12607,16 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 {
 	struct cfs_rq *cfs_rq;
 	struct sched_entity *se = &curr->se;
+	bool resched = false;
 
 	for_each_sched_entity(se) {
 		cfs_rq = cfs_rq_of(se);
-		entity_tick(cfs_rq, se, queued);
+		resched |= entity_tick(cfs_rq, se);
 	}
 
+	if (resched)
+		resched_curr(rq);
+
 	if (static_branch_unlikely(&sched_numa_balancing))
 		task_tick_numa(rq, curr);
 
-- 
2.31.1