Received-SPF: pass (google.com: domain of linux-kernel+bounces-191573-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
From: Ankur Arora <ankur.a.arora@oracle.com>
To: linux-kernel@vger.kernel.org
Cc: tglx@linutronix.de, peterz@infradead.org, torvalds@linux-foundation.org,
        paulmck@kernel.org, rostedt@goodmis.org, mark.rutland@arm.com,
        juri.lelli@redhat.com, joel@joelfernandes.org, raghavendra.kt@amd.com,
        sshegde@linux.ibm.com, boris.ostrovsky@oracle.com,
        konrad.wilk@oracle.com, Ankur Arora <ankur.a.arora@oracle.com>,
        Ingo Molnar <mingo@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>
Subject: [PATCH v2 29/35] sched: handle preempt=voluntary under PREEMPT_AUTO
Date: Mon, 27 May 2024 17:35:15 -0700
Message-Id: <20240528003521.979836-30-ankur.a.arora@oracle.com>
In-Reply-To: <20240528003521.979836-1-ankur.a.arora@oracle.com>
References: <20240528003521.979836-1-ankur.a.arora@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?fc0Rc9NCVRcDSu9vsp/Oid1ZufiGghcWvR2YJnmNebFkFE7hwQx2vBbCPKGy?=
 =?us-ascii?Q?+9ui7tNS7lQpTf/fuu2TpGkqKAp3TyPz+Ou8qEt4uqbZ9cw8w7T7qXK5i5ki?=
 =?us-ascii?Q?9LOcQUCnuX/5w10dKVX+i2EEo3r7NzaWPH03RJ2evavwmfiL69wgvm0SrXJ0?=
 =?us-ascii?Q?vK+9bhq2yqsAhefzAXznu8dtgyRN225TcChCljQGMEX8OwPmO6aYq0BhxwFB?=
 =?us-ascii?Q?YwwG25kCaeYvr53lM9nkddK9sBe6ZqRpd/mMxP5q+mP3F5SjCBlqdQM+f/q/?=
 =?us-ascii?Q?ZvtAjNm40A8JTgrMz14zFEEjZj8BS0m2C08vOU5sUMfI7IrT1ikuIk/XPbVP?=
 =?us-ascii?Q?U1x5OuTEPzRDSLYaecPxLHB1KeWnqlVzkQghZ3YkL1LaTsfI3bPK/PhlWIlt?=
 =?us-ascii?Q?g5vn/HenuZNf/VHt/7Z7jsmVdtLW3tm27mdmZ8Ez1Y3N7CMokNZy13zv7z71?=
 =?us-ascii?Q?vLmlB0N+4pEFl/UGGw6+/8Sbh/8I5qABzJMunPgwhL56I9F+RhkAG5jOuAXC?=
 =?us-ascii?Q?nH8XgbNhgk2OGWuZ0+OrroqsQm+haI/H0WM0xTOYO+qF1goFKfK+ep2Z77iE?=
 =?us-ascii?Q?/bi1kCTbSW/hPO6jlUJ+nYvnTwrf1dgDOH6f72jcVUk/uw6Hu2uX4fdUuHte?=
 =?us-ascii?Q?NU4ii5eeQIZqYRW0HM62VlD5I+JM2IryK7SnzAa5UgKX9FwaqvJhrZDNbgCg?=
 =?us-ascii?Q?e+ugatMIB9At04Inb6In71PF0bW5GVqd8nA67yF51dYmFYg7tsyg4/qMkv6+?=
 =?us-ascii?Q?x8mAA1/pR+ZN498UGiQEYc0jRgdCFre2vmKk1rpcTzGJmeKjtMk9nflDm0/k?=
 =?us-ascii?Q?+d2So6NsEjDGflN67agDRugjghA1NGI4vBWQPH9tuEjwMxrxy3+Y333Hzj+A?=
 =?us-ascii?Q?d7nC/9GN74czgu4XHDCrjImkxAmSmd0m0WtRcvXwXAy1yfiiI3RUIpg+qpSc?=
 =?us-ascii?Q?lAzh3n8+Uv3NqWNWjGEtIGg5FB3ikJdjeHj4I01U6XAvQ3Hi9aXcfb07ZDsM?=
 =?us-ascii?Q?2+GPgdm4fPEne1uHdhal020egYSBeacF+3nrxbcKroTHHPz2iuzsdkjMwKnG?=
 =?us-ascii?Q?J1RcTbEeBtQfTQtiNJGXoig/h9HfUuY1mhqQyqMnEBNlMVReLvSZ2IDNg8ZX?=
 =?us-ascii?Q?upIqg/lmwimEFnWjz2j7S4t3jPXdCgs1TldieY2xt1OyUxG5FRNKtu3+Tyxd?=
 =?us-ascii?Q?5N+DRtTyCTLk6CHzCzBzshdFrBNXPAK7Jo+PJL3nQrm6U9jVpxvaQIt3jrqp?=
 =?us-ascii?Q?ksaahGN5QkHdV2UZGoJwfsO4eVketqOKhNIozhh8XxAWFWWNy8D3v8le659a?=
 =?us-ascii?Q?bMhUAn0gi3MJs0mUd2zh+ppBUcJ5OS0umS/j7vaPUf2wtv/vz8ZKgZx3G9/P?=
 =?us-ascii?Q?aQPDvgHeVf6vjlg/kLAfGu1awrCUgySI6mnfyn4oBzRB3Xjv4/9UpgHZmQah?=
 =?us-ascii?Q?BgxbvorQ6RVU3PXI5AdvhfaJj24qfyZWb22WNg8A1YrbkKTfgxYHAspV7ES7?=
 =?us-ascii?Q?6ZqH7QPS1aMor0RGKbXAfFKKTRqYkpJz47RACa07McUnvpN6XzrsYSk0eGym?=
 =?us-ascii?Q?nsS/iBE5yLIn9ERb06hz2CEeojP46PB3bKLlge7MrpsUq4ZJ6PLPW4KEBErM?=
 =?us-ascii?Q?Lw=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	WxCfkZFA2Jm6zplXjgWEM3iT7OTVecs6erNOr8mVEDa0NpkefbGQRdLJvIz41WktoqalG5AVm+s/q2q3bhw+AAQhBSV2m0TR7HEDeNL/gV319Lm7uF3hGKHp3xi88Bp3Ba7WLs6ImSaa/yS/Nk7m3OKLjTHmwxBRBKCmPkqgQQ2gUUQ/wlM3OzTMWKeBSJb+TtZUaVjTAIA/uKtmHdOL4Xo/WswlrauCFJF4Q+pU2nCvcVom/wZkVDkbJzyVQdfCMuCxPNo1JEuR40BSL2eAruv/CIpgyhuLceeytWdpzf5TrtTAH86PfhBQzueZisbRKGtd7QOdsZlw6F5ssEBREiwb7SKBuBRJTqJStCEae4UvIoHzNTZGNiiqSBAad4sB9BUhWT8BHUmCcBzoKvZleQkK75Ypr55Hz07CBjHTDtiaidCvLwwllXy18NvfAzSoxNJRc9ChhpCOiDQBX55XqVv1PVtHX2VKmx/bqJ2N4iEkwYclnmUf0eT1zWubqp6g7SDX05pInP/bO8aCX9VLxjM9duggADc5PIX52LlEMuJL9IaZ3IFpeolz75U/h/Yfwmd9jwUOdE/pPWR1a/fIMTVvG/u7Fvy1NcLbBh9YYXA=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: fc4bd3c8-1971-4b9e-d8a1-08dc7eae442f
X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2024 00:36:51.4655
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: BFw70yCJqoal6X0eMmCQL3yM6fBRzXf4oOqjSWU0cylY12eMLoo2fQdGoMVcgR0CtWpEkn2cmalPw7A6aD1XcUPrRQmwhZJ2Z1qZkPw8rOY=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB7177
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16
 definitions=2024-05-27_06,2024-05-27_01,2024-05-17_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 suspectscore=0
 adultscore=0 bulkscore=0 malwarescore=0 mlxscore=0 spamscore=0
 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2405010000 definitions=main-2405280003
X-Proofpoint-GUID: T10g880XklQF7SQI3i1f6ijutrj9-P_P
X-Proofpoint-ORIG-GUID: T10g880XklQF7SQI3i1f6ijutrj9-P_P

The default preemption policy for voluntary preemption under
PREEMPT_AUTO is to schedule eagerly for tasks of higher scheduling
class, and lazily for well-behaved, non-idle tasks.

This is the same policy as preempt=none, with an eager handling of
higher priority scheduling classes.

Comparing a cyclictest workload with a background kernel load of
'stress-ng --mmap', shows that both the average and the maximum
latencies improve:

 # stress-ng --mmap 0 &
 # cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0 -q -D 300

                                     Min     (  %stdev )    Act     (  %stdev )   Avg     (  %stdev )   Max      (  %stdev )

  PREEMPT_AUTO, preempt=voluntary    1.73  ( +-  25.43% )   62.16 ( +- 303.39% )  14.92 ( +-  17.96% )  2778.22 ( +-  15.04% )
  PREEMPT_DYNAMIC, preempt=voluntary 1.83  ( +-  20.76% )  253.45 ( +- 233.21% )  18.70 ( +-  15.88% )  2992.45 ( +-  15.95% )

The table above shows the aggregated latencies across all CPUs.

Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Ziljstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Originally-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/lkml/87jzshhexi.ffs@tglx/
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 kernel/sched/core.c  | 12 ++++++++----
 kernel/sched/sched.h |  6 ++++++
 2 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index c25cccc09b65..2bc3ae21a9d0 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1052,6 +1052,9 @@ static resched_t resched_opt_translate(struct task_struct *curr,
 	if (preempt_model_preemptible())
 		return RESCHED_NOW;
 
+	if (preempt_model_voluntary() && opt == RESCHED_PRIORITY)
+		return RESCHED_NOW;
+
 	if (is_idle_task(curr))
 		return RESCHED_NOW;
 
@@ -2289,7 +2292,7 @@ void wakeup_preempt(struct rq *rq, struct task_struct *p, int flags)
 	if (p->sched_class == rq->curr->sched_class)
 		rq->curr->sched_class->wakeup_preempt(rq, p, flags);
 	else if (sched_class_above(p->sched_class, rq->curr->sched_class))
-		resched_curr(rq);
+		resched_curr_priority(rq);
 
 	/*
 	 * A queue event has occurred, and we're going to schedule.  In
@@ -8989,11 +8992,11 @@ static void __sched_dynamic_update(int mode)
 	case preempt_dynamic_none:
 		if (mode != preempt_dynamic_mode)
 			pr_info("%s: none\n", PREEMPT_MODE);
-		preempt_dynamic_mode = mode;
 		break;
 
 	case preempt_dynamic_voluntary:
-		preempt_dynamic_mode = preempt_dynamic_undefined;
+		if (mode != preempt_dynamic_mode)
+			pr_info("%s: voluntary\n", PREEMPT_MODE);
 		break;
 
 	case preempt_dynamic_full:
@@ -9003,9 +9006,10 @@ static void __sched_dynamic_update(int mode)
 
 		if (mode != preempt_dynamic_mode)
 			pr_info("%s: full\n", PREEMPT_MODE);
-		preempt_dynamic_mode = mode;
 		break;
 	}
+
+	preempt_dynamic_mode = mode;
 }
 
 #endif /* CONFIG_PREEMPT_AUTO */
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 107c5fc2b7bb..ee8e99a9a677 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2468,6 +2468,7 @@ enum resched_opt {
 	RESCHED_DEFAULT,
 	RESCHED_FORCE,
 	RESCHED_TICK,
+	RESCHED_PRIORITY,
 };
 
 extern void __resched_curr(struct rq *rq, enum resched_opt opt);
@@ -2482,6 +2483,11 @@ static inline void resched_curr_tick(struct rq *rq)
 	__resched_curr(rq, RESCHED_TICK);
 }
 
+static inline void resched_curr_priority(struct rq *rq)
+{
+	__resched_curr(rq, RESCHED_PRIORITY);
+}
+
 extern void resched_cpu(int cpu);
 
 extern struct rt_bandwidth def_rt_bandwidth;
-- 
2.31.1