Received-SPF: pass (google.com: domain of linux-kernel+bounces-62981-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
From: Ankur Arora <ankur.a.arora@oracle.com>
To: linux-kernel@vger.kernel.org
Cc: tglx@linutronix.de, peterz@infradead.org, torvalds@linux-foundation.org,
        paulmck@kernel.org, akpm@linux-foundation.org, luto@kernel.org,
        bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com,
        mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org,
        willy@infradead.org, mgorman@suse.de, jpoimboe@kernel.org,
        mark.rutland@arm.com, jgross@suse.com, andrew.cooper3@citrix.com,
        bristot@kernel.org, mathieu.desnoyers@efficios.com,
        geert@linux-m68k.org, glaubitz@physik.fu-berlin.de,
        anton.ivanov@cambridgegreys.com, mattst88@gmail.com,
        krypton@ulrich-teichert.org, rostedt@goodmis.org,
        David.Laight@ACULAB.COM, richard@nod.at, mjguzik@gmail.com,
        jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com,
        boris.ostrovsky@oracle.com, konrad.wilk@oracle.com,
        Ankur Arora <ankur.a.arora@oracle.com>
Subject: [PATCH 21/30] sched: schedule eagerly in resched_cpu()
Date: Mon, 12 Feb 2024 21:55:45 -0800
Message-Id: <20240213055554.1802415-22-ankur.a.arora@oracle.com>
In-Reply-To: <20240213055554.1802415-1-ankur.a.arora@oracle.com>
References: <20240213055554.1802415-1-ankur.a.arora@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?EMosoI7PFq1cO8twuAmOxV2baxrtE1XLQqUgH+ECIcZXjNr76OBzOtoEg/Pv?=
 =?us-ascii?Q?1qYi9BnS9m5tVgHY9motCpfQYsFWE4kuzu+mp6rx9k0tt0ORjpRjFrMarmLw?=
 =?us-ascii?Q?bJIWn7XwwNBYbCT12kLkYPRYPF86yva/7S8IRzOKYfI+KKlDJuRVwzOR3L8Y?=
 =?us-ascii?Q?dFLavr/yv7ujX62Ekuv8mTKoa0Xdk6vij9TQqOUIIYKJ0JmyFjsod1LTTmvM?=
 =?us-ascii?Q?A95KyJ72BGb5fKEAIH25BEEhkJQYV9TXJ18zucUL75E0KlS5qri1WTupGpdc?=
 =?us-ascii?Q?EZrJt/sUSmkozR+PEn06Id5ZD2740V9ikt1XIToR3YRpf59K0j4QN+3ITeuQ?=
 =?us-ascii?Q?TtLf8/iZo5KTY3FFTU6mP4QWzIORN6ZhR0ZG5dMNVpVtch+SKzGzg1bH5YVc?=
 =?us-ascii?Q?V18FS615iIEwJWHOAoK+Vm3krVmse+Fc04XihHO9zbdRitWMtsDrIRHym5mu?=
 =?us-ascii?Q?bvsL17wsq2ro6qZFQCwh5vKCgZe5i92sO7nRAu23iAgIstRmpsCVhgR8gT0G?=
 =?us-ascii?Q?oukWt90J7zDJyQrTYjL8biJd79FVBh4Y0wx5X3KlBJClSTrvlyW9uScbVpau?=
 =?us-ascii?Q?2AscS4EhyGmkyKbE3p6cSr/ONa5lNDGyNZuMBKWUaeer7RAzhdssWgb4WQ1Y?=
 =?us-ascii?Q?Kom6gm93zuhHVwt/+cfumrDJHo7/yNj3nn2dr9zcH/HbMkmNmcATMYKf+3aH?=
 =?us-ascii?Q?HlLZSc0Ps/WCCE1MUy/W+uIKHiUVeLS9xDn37rEODUG+FfO+/dt8IPJmJ7mg?=
 =?us-ascii?Q?kDX5v5W3OwjgIVQeQcpWblZXuoV2aN/6iPHT40W/WCC9cMwwGuG2OkaQpiED?=
 =?us-ascii?Q?N960mrqR08scAcWTy3/zsawy0wQHWv71tbIgikPwm8P8Q55X9i7r8DtNgwMZ?=
 =?us-ascii?Q?AZFZ2cxm6HsgH/N0SODTyk999DCIuU2jSI0h6muqMdyF/xwJ+cePWrQcb6Y5?=
 =?us-ascii?Q?XxCDTMGd59LzMX1ZmDbSCuQoPeEEzW0hnRlsImrrui7GjHT6+IF6N/UDC7ce?=
 =?us-ascii?Q?EdjfbhfnksxRprF6j8dZth/BAd6ZRt3N13MN+M2UVhV4ORxfcjnbqzZtiobC?=
 =?us-ascii?Q?j2OWJBxD2kryAQH75DnY7JGyLBOqAx+eWOga4CpPCcr2YnOv/VSm5nTggLaD?=
 =?us-ascii?Q?AB7hTrNmD4foxlqytVBXKlzGQ7cXv7W+tho+qdE3Y9dMi8KKvkZWJUDsJ9WB?=
 =?us-ascii?Q?wEeKJtMFpX/rtzL1KsOquNSbXRE1+lzT8AkYkV7Cl9kKsJaMxUVM6+dfmhkU?=
 =?us-ascii?Q?7vi7bfiooQo+yywGSZxON8wyvQGoLdIQ0R1NiTL/ymwMTNFnZ8FqyGo7TTgw?=
 =?us-ascii?Q?XJIkd3C6JnhYGnpteKjB8eu4EsmyW8c9jeA/kXH4HBgY6W99ikU1l26Jie8Z?=
 =?us-ascii?Q?LwLQH0as+1sbRcm+TXfkMpDypZn/1Jq8QFSf3iyG1/J7Znm82LKMfcf2tPrJ?=
 =?us-ascii?Q?fwpuX8hsOPhEdEBoORsIVA/Ocu5VsokqmP3a8vQXn5NHZdV2eqAFuHZR04Sg?=
 =?us-ascii?Q?St6uxfwtumofz8SzvinsKkpgh1aK9YFowY3Vo1XZWrXRQgjefMIpFxCHWLkV?=
 =?us-ascii?Q?b7TkmlpnpsWWo59cbvltLidlAPkHWAIBnpU2eR0/s+TD2V2ejxaI7IU9rkzI?=
 =?us-ascii?Q?YA=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	uvgLDQwRrp78BvzFju4XRaGBeqzrLTP32DnZ0LratToHAnaCEn03s5q8284gz/bd973bwhZZOx5YHlZXuc89uyYB2eO07jrHEl8CrZAEDgPmT5kfD66qGYQ7BY8pHgGKuID8TRaxlBhH1RMDNK7UbisgyAnoU6ts8ct94fWOlLw2z3wCfeQavdkKM/kpjSlV30tg0NvqTLH/TcV7TuaJZxHMf52MDXOs4X1n3aVE1f4/9ImoU/0ePUO5aREt9Z5IIHJUaYMqugvrwn1jtos7IwR8eCzG69WmZXVVQUTnTkjH0dTYPBkguYDm+fWSPtcdBgfU30JmysiXGcZ9nRWnpp4zeZxQ3P+kxXgPbedj8aa9Xm7ra7LTMT7no/37P+2qM8/H81rOBZbMvdgokhKfq4M8e9cHX7Q6ZxY5QT0ghcykpeIpdt3yXdZuaHXKmBOyLYlGLrStAnfvLJtmM42wyPk8ES/XmLQTBEy7LJYbyN8vdRG60BYnBNLsAeSH/8NskGEhbBoHXt4wXzQ+AynrSkUuJG19QW/pwzNEG6jMld2B+FhUbl+p1y8J1I9IjRRxir9GDOnYRjYWU9MEWSwufKW0k9dedsKYAZDDNhdMGTM=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 62d67274-e41b-43dd-4fc0-08dc2c588a4b
X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Feb 2024 05:56:37.0147
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 9Bhp3U08bZ40yMG+a3h4JGY0c2Fi1201NmTUD5zQYcql83HejxX1WA42fxFVWtWL8gPz1OaFoQ5a0jtI7RZujYFqIJungYJRM1TFRVSo950=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB6906
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-02-13_02,2024-02-12_03,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0
 mlxlogscore=999 bulkscore=0 phishscore=0 mlxscore=0 suspectscore=0
 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2402130043
X-Proofpoint-GUID: utIk-zCDd7SrAha4fZjevqfXPB6NuRcy
X-Proofpoint-ORIG-GUID: utIk-zCDd7SrAha4fZjevqfXPB6NuRcy

resched_cpu() is used as an RCU hammer of last resort. Force
rescheduling eagerly with tif_resched(NR_now).

Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 kernel/sched/core.c  | 15 +++++++++++----
 kernel/sched/sched.h |  1 +
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 5e3dd95efb73..de963e8e2bee 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1035,9 +1035,9 @@ void wake_up_q(struct wake_q_head *head)
  * For preemption models other than PREEMPT_AUTO: always schedule
  * eagerly.
  *
- * For PREEMPT_AUTO: allow everything, whether running in user or
- * kernel context, to finish its time quanta, and mark for
- * rescheduling at the next exit to user.
+ * For PREEMPT_AUTO: schedule idle threads eagerly, allow everything
+ * else, whether running in user or kernel context, to finish its time
+ * quanta, and mark for rescheduling at the next exit to user.
  */
 static resched_t resched_opt_translate(struct task_struct *curr,
 				       enum resched_opt opt)
@@ -1045,6 +1045,9 @@ static resched_t resched_opt_translate(struct task_struct *curr,
 	if (!IS_ENABLED(CONFIG_PREEMPT_AUTO))
 		return NR_now;
 
+	if (opt == RESCHED_FORCE)
+		return NR_now;
+
 	if (is_idle_task(curr))
 		return NR_now;
 
@@ -1106,7 +1109,11 @@ void resched_cpu(int cpu)
 
 	raw_spin_rq_lock_irqsave(rq, flags);
 	if (cpu_online(cpu) || cpu == smp_processor_id())
-		resched_curr(rq);
+		/*
+		 * resched_cpu() is typically used as an RCU hammer.
+		 * Mark for imminent resched.
+		 */
+		__resched_curr(rq, RESCHED_FORCE);
 	raw_spin_rq_unlock_irqrestore(rq, flags);
 }
 
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index c3ae70ad23ec..3600a8673d08 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2464,6 +2464,7 @@ extern void reweight_task(struct task_struct *p, int prio);
 
 enum resched_opt {
 	RESCHED_DEFAULT,
+	RESCHED_FORCE,
 };
 
 extern void __resched_curr(struct rq *rq, enum resched_opt opt);
-- 
2.31.1