Received-SPF: pass (google.com: domain of linux-kernel+bounces-62962-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
From: Ankur Arora <ankur.a.arora@oracle.com>
To: linux-kernel@vger.kernel.org
Cc: tglx@linutronix.de, peterz@infradead.org, torvalds@linux-foundation.org,
        paulmck@kernel.org, akpm@linux-foundation.org, luto@kernel.org,
        bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com,
        mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org,
        willy@infradead.org, mgorman@suse.de, jpoimboe@kernel.org,
        mark.rutland@arm.com, jgross@suse.com, andrew.cooper3@citrix.com,
        bristot@kernel.org, mathieu.desnoyers@efficios.com,
        geert@linux-m68k.org, glaubitz@physik.fu-berlin.de,
        anton.ivanov@cambridgegreys.com, mattst88@gmail.com,
        krypton@ulrich-teichert.org, rostedt@goodmis.org,
        David.Laight@ACULAB.COM, richard@nod.at, mjguzik@gmail.com,
        jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com,
        boris.ostrovsky@oracle.com, konrad.wilk@oracle.com,
        Ankur Arora <ankur.a.arora@oracle.com>
Subject: [PATCH 09/30] sched: __schedule_loop() doesn't need to check for need_resched_lazy()
Date: Mon, 12 Feb 2024 21:55:33 -0800
Message-Id: <20240213055554.1802415-10-ankur.a.arora@oracle.com>
In-Reply-To: <20240213055554.1802415-1-ankur.a.arora@oracle.com>
References: <20240213055554.1802415-1-ankur.a.arora@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?DHP7XuVuCq0lBPQ4UuesfaOizcu8K2u5E1dO8HvLX2jiGC4oZxV0MjRKsWEt?=
 =?us-ascii?Q?wHinddM6DXFY5gg/ijGn60giILpBt2ABSJ/kxeDmebjs9pkKBtLa43NJ1i+n?=
 =?us-ascii?Q?4Lh21vgh56ha6eKGJkUOoXKoDsCJ0sF0Zo1YqfgHQ4T5+C5z6hQZe4TQtM09?=
 =?us-ascii?Q?Ja3lWDdYv3DYUT+nnpHAzmrMf+BiFR+WrHaDHuWLQ9xpy3OLveqEGFVl+l5V?=
 =?us-ascii?Q?7+NB0ZfXjxxJE8GGiOusT11wkz9tL3pa4GEBHIRzlUI3Z+aRztdwnc655sJl?=
 =?us-ascii?Q?YrQst+Kppb/QwONEQnSla0w/5d+IJHpcRzoQLi8XzD9MiDRyX+dWbwUqz5md?=
 =?us-ascii?Q?e+ZzGciKOCAnbctHlmEqtbVYSfWliho6zU+oUajtkw9V89vLLYnjAUzwj+oj?=
 =?us-ascii?Q?Zl07Qbwtiim/sCCANZEla4BrrGu6Bbr9YZM7GzG4pZLV/kvhP2sExixBPGRA?=
 =?us-ascii?Q?Hw8aGca4Phe55ymGXxLM+zygvY+1StoPjc03VoaJBi186Likbs4z8QPzw41d?=
 =?us-ascii?Q?Q9CSP29I7/OIuuzklLGBYkifdZJsa1XMQolxGB2VmAtUrbkEGeM81FqZvoK+?=
 =?us-ascii?Q?8JDHOz2VcLXG/+mMfCK29I92XR1Ixw5J7xBL7RtInViUsJOq3TrNL1RYwfxL?=
 =?us-ascii?Q?/K9/TAkxmlFICXiDCssCmqxpS2yOaTo+EEW1WPA+44UqrWsXtEwePlL/i2uw?=
 =?us-ascii?Q?wlE+CuJQHmNy/HM0EaU249MZKz4SPsHpq6d5HnH3Kb1Iefb/iWmkkv9SyvIA?=
 =?us-ascii?Q?802O8BYsGJ3DvS8A3YqHGrSMVGZNc36nR20ooozgOHx2GiZPgzLd+4VEveS6?=
 =?us-ascii?Q?f6JFpOQfdrifN5ZuM1P8BA8OZWPeJiBI88r7ldhT6yYgcBSlXwHxyNncS6J3?=
 =?us-ascii?Q?kUchCq/XCkn6GsMB8Str+dk/cmtPlmSJ39ERePcGN5cOuj1Ga7ZyM1Or+CGf?=
 =?us-ascii?Q?hkY18rN7OYwKHQIfb52KrIABYVsZLCXuQx+/IMcwzDdu7UMm03Rvpx+DwCeL?=
 =?us-ascii?Q?GKcndxulpfajTZnhz03nBM6kRS5bVoPG+UKBdP0TX0cwJWiuEmsQngI6CwsX?=
 =?us-ascii?Q?5AFOiuHyo2f0yrVJx+m1oR/POoPFc2ZG0wlSuaLCLIujlelBxZtOQm4+seYI?=
 =?us-ascii?Q?qx9R8TDXFzGu/H9RK9VtiCrWPm24gozRboKm2m1v1VQ2m6eZCwCSMsuPigia?=
 =?us-ascii?Q?YKh7cmVwNO1pi+hdjtHXk0MvMoAQfvtBDPyFQ1DyxZlg7XBmvI1KptouHBud?=
 =?us-ascii?Q?r9DvzUEgFjnxdTLt6CNnTpUA0zXuBMP2Z4vEvATvo9ucppT42O+WSnqkzS8S?=
 =?us-ascii?Q?3PMBBXSslFQn6xeClj/yUZCLOUs9f5YBWs3a1sXKCkgzmP9SwZhcs5qA9t5E?=
 =?us-ascii?Q?sa0kTue5wxcL8xGGUvUdJh1XqxbCwlsMzveG4t353mB4rqxlbFTMsldUORna?=
 =?us-ascii?Q?xxhkoVhDJ73UZ5BrYkVrOPUui6XBE6LBVnELElmc/b7meZfxdtdryM78BEQ4?=
 =?us-ascii?Q?0LXXoX6UQ1VQo9upDLXXU3xKKBgZrfTAfpdnY0EMDjf2WVNElZi1m5C1j2/o?=
 =?us-ascii?Q?qQA02YQF51eYlH/Zl9JSBNeb7V++2KOs0/ATWA4ZQfGY0yZVKRh/RkQsoQnW?=
 =?us-ascii?Q?yw=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	W9ShxVOdxY3UIQ9q0vjaNLlfrGjlqRb6FSnzOyfZDsIPNnqgeF7WAbRw71iMrbzrd8nyXQkn33oufqO+YB8FwJYLDXrSOPNiY6EuXZ+R8mO4RkM8d3VLA8hxxUnjohfHvR++e3z6DtmZHd2GZ2Dusrrlkf+ze1VIrvkVXNRVaNIxpqhJHR/RW0vAtRv1eHt/giSBppH0Ey5vKxKhAGmnfrFajq4WcsOKbZhOZAc3dPlfwjqcVbcsUVQyunvUo1dEp0MOwJuI0C8ot4fHun954zeZO9dJsTqjsWglsE5AKrKN48vZ2sfxx3iq6oR1D2LOZsEFHQ7EwmVkbB2zAzhRy3GzNDgAdTCjcwwOjroXOiQLohRwkBHVJLhsqQw7j5sXXP84NH1zQNmNxIuaqIs5QKilhvWMGDnCRqOjxGzYg5tyY0kHGg6uGSqO33lhILbBckRG3lGB7dKsseodVcogPvkPOLNmCWI+ZaCp6g8FRdMP15We8421MXqrGWzXDINSDma0kIhFqQesD/R89gqfKFKb6phhpbnZbv/+Ya6KSneHzpbEJlPba7IERgfYh38q45sGDAvixshTTim1J/r6Jl1eCgTnqudhgMCW5fQwJ1A=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a1edaddc-6855-4a5b-e3ea-08dc2c587585
X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Feb 2024 05:56:02.1458
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: yYnHackW7j9ejiNcZVoaicbLHDvJ2owoCSj2JRePtoBNpNkMl+gAvfqgzD+RuDVl8KeEg1lLX9UGJtIrWgHZyjlFHY7xrxZGHq+sx2V/XZo=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4642
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-02-13_02,2024-02-12_03,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0
 mlxlogscore=540 bulkscore=0 phishscore=0 mlxscore=0 suspectscore=0
 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2402130043
X-Proofpoint-GUID: W1PZE2iODLlVkADUtjFxfwW_DvZYUoqN
X-Proofpoint-ORIG-GUID: W1PZE2iODLlVkADUtjFxfwW_DvZYUoqN

Various scheduling loops recheck need_resched() to avoid a missed
scheduling opportunity.

Explicitly note that we don't need to check for need_resched_lazy()
since that only needs to be handled at exit-to-user.

Also update the comment above __schedule() to describe
TIF_NEED_RESCHED_LAZY semantics.

Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Originally-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/lkml/87jzshhexi.ffs@tglx/
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 kernel/sched/core.c | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 41c3bd49a700..8e492d20021c 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6573,18 +6573,21 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
  *
  *   1. Explicit blocking: mutex, semaphore, waitqueue, etc.
  *
- *   2. TIF_NEED_RESCHED flag is checked on interrupt and userspace return
- *      paths. For example, see arch/x86/entry_64.S.
+ *   2. TIF_NEED_RESCHED flag is checked on interrupt and TIF_NEED_RESCHED[_LAZY]
+ *      flags on userspace return paths. For example, see kernel/entry/common.c
  *
- *      To drive preemption between tasks, the scheduler sets the flag in timer
- *      interrupt handler scheduler_tick().
+ *      To drive preemption between tasks, the scheduler sets one of the need-
+ *      resched flags in the timer interrupt handler scheduler_tick():
+ *        - !CONFIG_PREEMPT_AUTO: TIF_NEED_RESCHED.
+ *        - CONFIG_PREEMPT_AUTO: TIF_NEED_RESCHED or TIF_NEED_RESCHED_LAZY
+ *          depending on the preemption model.
  *
  *   3. Wakeups don't really cause entry into schedule(). They add a
  *      task to the run-queue and that's it.
  *
  *      Now, if the new task added to the run-queue preempts the current
- *      task, then the wakeup sets TIF_NEED_RESCHED and schedule() gets
- *      called on the nearest possible occasion:
+ *      task, then the wakeup sets TIF_NEED_RESCHED[_LAZY] and schedule()
+ *      gets called on the nearest possible occasion:
  *
  *       - If the kernel is preemptible (CONFIG_PREEMPTION=y):
  *
@@ -6802,6 +6805,11 @@ static __always_inline void __schedule_loop(unsigned int sched_mode)
 		preempt_disable();
 		__schedule(sched_mode);
 		sched_preempt_enable_no_resched();
+
+		/*
+		 * We don't check for need_resched_lazy() here, since it is
+		 * always handled at exit-to-user.
+		 */
 	} while (need_resched());
 }
 
@@ -6907,7 +6915,7 @@ static void __sched notrace preempt_schedule_common(void)
 		preempt_enable_no_resched_notrace();
 
 		/*
-		 * Check again in case we missed a preemption opportunity
+		 * Check again in case we missed an eager preemption opportunity
 		 * between schedule and now.
 		 */
 	} while (need_resched());
-- 
2.31.1