Received-SPF: pass (google.com: domain of linux-kernel+bounces-62979-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: Ankur Arora <ankur.a.arora@oracle.com>
To: linux-kernel@vger.kernel.org
Cc: tglx@linutronix.de, peterz@infradead.org, torvalds@linux-foundation.org,
        paulmck@kernel.org, akpm@linux-foundation.org, luto@kernel.org,
        bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com,
        mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org,
        willy@infradead.org, mgorman@suse.de, jpoimboe@kernel.org,
        mark.rutland@arm.com, jgross@suse.com, andrew.cooper3@citrix.com,
        bristot@kernel.org, mathieu.desnoyers@efficios.com,
        geert@linux-m68k.org, glaubitz@physik.fu-berlin.de,
        anton.ivanov@cambridgegreys.com, mattst88@gmail.com,
        krypton@ulrich-teichert.org, rostedt@goodmis.org,
        David.Laight@ACULAB.COM, richard@nod.at, mjguzik@gmail.com,
        jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com,
        boris.ostrovsky@oracle.com, konrad.wilk@oracle.com,
        Ankur Arora <ankur.a.arora@oracle.com>
Subject: [PATCH 22/30] sched/fair: refactor update_curr(), entity_tick()
Date: Mon, 12 Feb 2024 21:55:46 -0800
Message-Id: <20240213055554.1802415-23-ankur.a.arora@oracle.com>
In-Reply-To: <20240213055554.1802415-1-ankur.a.arora@oracle.com>
References: <20240213055554.1802415-1-ankur.a.arora@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?iEZi7uGslFmOkVWCV2Hi45kmAQL6GS4UxfsPQO9ZDW5px9wCGd68B/SGXy9O?=
 =?us-ascii?Q?l2MOLmH1toU6UHnBJ/5LJH/wK+rs0RdGeTlMxAf65gadDi5Xdpqf8aKRshg1?=
 =?us-ascii?Q?rmawGZUmxAzjATtKVmQBrZ76HozfJ+c9CmVdwQf/B5vB9pgzGogvomG56f/h?=
 =?us-ascii?Q?z6+3X+OWhZLrEJoR2P3c5t1dLLCTQPVpHqoAi3lW37GRv1xPA0uTUHHIc308?=
 =?us-ascii?Q?90/ciqHROtaYZbSBoWENagVAxYVAcjmxBRsjvUKoUcmZN1uhK3aiBa9a822M?=
 =?us-ascii?Q?zQfzsf5I6/y6frTGPCGd23FNJbMR0K95oEr5uqYHGtkW7WAZuEVOPqFD+ts0?=
 =?us-ascii?Q?xDiuaPrVjFbG5PYs3fq0LcNRIwR2Vb+2JPtapTzY1EP2E17OPwKqE8yp6tL4?=
 =?us-ascii?Q?2d+zEUSJW4vOmfhkxCqbH5KsKpIZRbOEIZ6Be4VXSh2qs9B4fRyfi+EYeQ7x?=
 =?us-ascii?Q?xNWILA663q8I9VgTOjFF/U2giVxnMZ0evEDPuB2hcLsSFGJg4Pp3KJd+6MZe?=
 =?us-ascii?Q?t3ynunk0DTzm0DgvHjV0p9wASIFD0uemh8CpgpBZ+PHMJL4+t5dzYoMrE/Bw?=
 =?us-ascii?Q?rDyHMuRUgJeThbQFIL4qedydcmeOn1UYRDZZJ3wjtmUm9AgBCOmdrBpygeSN?=
 =?us-ascii?Q?5jYqnS7+WBk7tgiiOf8bJyAFAHQ7RwiO+3y0ZkkQT+gTZj5S4Hxz0VTPpuIm?=
 =?us-ascii?Q?Cp6udx714s+f0AN7UkI5fTsomLl+yWtE4L7Xin1AxZliZKNh9kzhdIPeSYnw?=
 =?us-ascii?Q?bm5tl+krqWvFGEJVi1k5RsmO9fcBPupWBMMHg7ppWMtlH4bo6X1jVEeM4ihw?=
 =?us-ascii?Q?gRtI71FZcwEuYFKATdlWflGJOYn2K7T0XoabkUp2mynslB/Rol3s5BT2qlCC?=
 =?us-ascii?Q?W/hBGdkgC15DGPlQkXSVKnuLyqLWags60voAZ3XDC35CLNtvyJtltLpfIYa7?=
 =?us-ascii?Q?kiklRZm3+IAWxQPwx4SNPPRMG9775sL34nt7H9DhiVl8acwVLvl7chjVLN7B?=
 =?us-ascii?Q?vCQstrp9HPg28xedfjK1ReaArzo3O3cxM/rzLTm5k2vnp0DTXYkvh/2ohWHx?=
 =?us-ascii?Q?IuaVRcI78XjvPxA779RC+Q34x9Lr84H0i9GW0yOZsUyIW70l88pQUMFRTi4X?=
 =?us-ascii?Q?oTfO4X826+OUnruo1lAdzV7xd5bli3gz2H2zSln1OTPouqsVtyrw1BV7uuT9?=
 =?us-ascii?Q?ZntQ+wIuDuQMqZQRYVRaNpvpL5R/oV5QFFSWcE5W9wTkpllPBV6J1piso+FT?=
 =?us-ascii?Q?CTJXrZ+omqzXs3PopJDVaJatE3MlwNoYYC/xzmcmTNyZsy242Xv4PzBh86rV?=
 =?us-ascii?Q?PDwpAjIMiJoVDqoIbr1ITazENQk5y0f81YUXIxZQXcZFu0SxLIlFdyWbR6LC?=
 =?us-ascii?Q?/+C+zoFyU2BRJMxHYWrrwqhJSkieHX8YubPdIlMKtMcDqFn6arQPLxv+xv59?=
 =?us-ascii?Q?VSUntwcWEFjbC1dJ467BGAEXM3OeNIZiym3rLocdLaiykI8nhvyt2j8+5Pik?=
 =?us-ascii?Q?LpfPvqExx4qGOmPlN8STFnk0Nvhse6pYFez3R5u00H4dvz86yH+8dKb4j35r?=
 =?us-ascii?Q?HHXkDzO82oHWIEOkNETehZwVKB/wdx8GARNmfVWWsiE08GeNgZYBSRsoMl+S?=
 =?us-ascii?Q?Tg=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	mkJNLTLXgJMFiUFXR7xs57U1ooJNsqw6CvUQG1fFj0lZ4tuXygAq9lTco+lT23lSz9UvDl7a0r/y0bsquBE7DENYaH7mdX4FGDdh/W2TrL1Kx0/I8RX9cGiMmNyTfUDbz5/9lNytlRDrQ5NrvXlc4zHB/+QMZKiKeOW1eFIB1G6Cd5bLckaSJRV4DcPhTllGn3/3K9mq1RlfnUUKlGL1wTJAPZh0xcbXiulxQRP/qWNNyMEuVLrNReBfckRdtbeU+NUfS31YD6H0I/XvMc/pzk8wUNdpDMyw4r2hClrA7qaLeXh9UR704lq3dwTK8gzRxYY3Dk4b9IXiUVsthpw0fGWHydSMDAhe9cl+Q7wBMCv2wO5z1Mo1hGeN/um2wQMecA3VOlPIksm7x2ewrq6uvqgoT8YSQEAsGm3/k87Q0qcgCcSuoFO5YkbxP1AEUJr9AzQD+T5yYaFosgNiekpXZKAGlRYK/QRlaf1exHp/8EFaFOKpgDf/FzD7N+W2gQhC0lj8cpcP6y0E14spZf0N4hoMDUMKwnFYOhWuMAqvxdF5RZz+0R1WkJRwagW7j5nN/sen86xxhWZZ+NZGKCAsHDDuyGz6WwWXG75qhJhVF3g=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e95b355e-8ebb-4a54-4956-08dc2c588b5f
X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Feb 2024 05:56:38.8141
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: P4ujcLPPrnHArHoIid3mQNREUFDIApEe++vvlaz2udrgUdO5jBpoEWBUibpDDtFiURNe6Zp3yKwKFS4UIUA15RmPEY3a2Zttg0b8e7nldgo=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB6906
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-02-13_02,2024-02-12_03,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0
 mlxlogscore=999 bulkscore=0 phishscore=0 mlxscore=0 suspectscore=0
 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2402130043
X-Proofpoint-ORIG-GUID: 0RdEmUMVz6onagjUqz27alULBnpRVvBV
X-Proofpoint-GUID: 0RdEmUMVz6onagjUqz27alULBnpRVvBV

When updating the task's runtime statistics via update_curr()
or entity_tick(), we call resched_curr() to resched if needed.

Refactor update_curr() and entity_tick() to only update the stats
and deferring any rescheduling needed to task_tick_fair() or
update_curr().

Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Suggested-by: Peter Ziljstra <peterz@infradead.org>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 kernel/sched/fair.c | 54 ++++++++++++++++++++++-----------------------
 1 file changed, 27 insertions(+), 27 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index ae9b237fa32b..278eebe6656a 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -975,10 +975,10 @@ static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se);
  * XXX: strictly: vd_i += N*r_i/w_i such that: vd_i > ve_i
  * this is probably good enough.
  */
-static void update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
+static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
 	if ((s64)(se->vruntime - se->deadline) < 0)
-		return;
+		return false;
 
 	/*
 	 * For EEVDF the virtual time slope is determined by w_i (iow.
@@ -996,9 +996,11 @@ static void update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
 	 * The task has consumed its request, reschedule.
 	 */
 	if (cfs_rq->nr_running > 1) {
-		resched_curr(rq_of(cfs_rq));
 		clear_buddies(cfs_rq, se);
+		return true;
 	}
+
+	return false;
 }
 
 #include "pelt.h"
@@ -1153,26 +1155,35 @@ s64 update_curr_common(struct rq *rq)
 /*
  * Update the current task's runtime statistics.
  */
-static void update_curr(struct cfs_rq *cfs_rq)
+static bool __update_curr(struct cfs_rq *cfs_rq)
 {
 	struct sched_entity *curr = cfs_rq->curr;
 	s64 delta_exec;
+	bool resched;
 
 	if (unlikely(!curr))
-		return;
+		return false;
 
 	delta_exec = update_curr_se(rq_of(cfs_rq), curr);
 	if (unlikely(delta_exec <= 0))
-		return;
+		return false;
 
 	curr->vruntime += calc_delta_fair(delta_exec, curr);
-	update_deadline(cfs_rq, curr);
+	resched = update_deadline(cfs_rq, curr);
 	update_min_vruntime(cfs_rq);
 
 	if (entity_is_task(curr))
 		update_curr_task(task_of(curr), delta_exec);
 
 	account_cfs_rq_runtime(cfs_rq, delta_exec);
+
+	return resched;
+}
+
+static void update_curr(struct cfs_rq *cfs_rq)
+{
+	if (__update_curr(cfs_rq))
+		resched_curr(rq_of(cfs_rq));
 }
 
 static void update_curr_fair(struct rq *rq)
@@ -5487,13 +5498,13 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
 	cfs_rq->curr = NULL;
 }
 
-static void
-entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
+static bool
+entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
 {
 	/*
 	 * Update run-time statistics of the 'current'.
 	 */
-	update_curr(cfs_rq);
+	bool resched = __update_curr(cfs_rq);
 
 	/*
 	 * Ensure that runnable average is periodically updated.
@@ -5501,22 +5512,7 @@ entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
 	update_load_avg(cfs_rq, curr, UPDATE_TG);
 	update_cfs_group(curr);
 
-#ifdef CONFIG_SCHED_HRTICK
-	/*
-	 * queued ticks are scheduled to match the slice, so don't bother
-	 * validating it and just reschedule.
-	 */
-	if (queued) {
-		resched_curr(rq_of(cfs_rq));
-		return;
-	}
-	/*
-	 * don't let the period tick interfere with the hrtick preemption
-	 */
-	if (!sched_feat(DOUBLE_TICK) &&
-			hrtimer_active(&rq_of(cfs_rq)->hrtick_timer))
-		return;
-#endif
+	return resched;
 }
 
 
@@ -12617,12 +12613,16 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 {
 	struct cfs_rq *cfs_rq;
 	struct sched_entity *se = &curr->se;
+	bool resched = false;
 
 	for_each_sched_entity(se) {
 		cfs_rq = cfs_rq_of(se);
-		entity_tick(cfs_rq, se, queued);
+		resched |= entity_tick(cfs_rq, se);
 	}
 
+	if (resched)
+		resched_curr(rq);
+
 	if (static_branch_unlikely(&sched_numa_balancing))
 		task_tick_numa(rq, curr);
 
-- 
2.31.1