Received-SPF: pass (google.com: domain of linux-kernel+bounces-191552-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
From: Ankur Arora <ankur.a.arora@oracle.com>
To: linux-kernel@vger.kernel.org
Cc: tglx@linutronix.de, peterz@infradead.org, torvalds@linux-foundation.org,
        paulmck@kernel.org, rostedt@goodmis.org, mark.rutland@arm.com,
        juri.lelli@redhat.com, joel@joelfernandes.org, raghavendra.kt@amd.com,
        sshegde@linux.ibm.com, boris.ostrovsky@oracle.com,
        konrad.wilk@oracle.com, Sean Christopherson <seanjc@google.com>,
        Valentin Schneider <valentin.schneider@arm.com>,
        Marco Elver <elver@google.com>,
        Frederic Weisbecker <frederic@kernel.org>,
        David Matlack <dmatlack@google.com>,
        Friedrich Weber <f.weber@proxmox.com>,
        Ankur Arora <ankur.a.arora@oracle.com>, Chen Yu <yu.c.chen@intel.com>
Subject: [PATCH v2 02/35] sched/core: Drop spinlocks on contention iff kernel is preemptible
Date: Mon, 27 May 2024 17:34:48 -0700
Message-Id: <20240528003521.979836-3-ankur.a.arora@oracle.com>
In-Reply-To: <20240528003521.979836-1-ankur.a.arora@oracle.com>
References: <20240528003521.979836-1-ankur.a.arora@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?5Ao4JeuVZSr1MxF15q1Ol3quj/oh4wOtYJwAaPDtGjduLifPfckPX6YYJcnA?=
 =?us-ascii?Q?ZqX9vNqeofZKFFCnvGqB0Ziost8HRPxnWHehlru3LvVlN0DK5yGTL3y7JJXX?=
 =?us-ascii?Q?ohHNfbhrj3l0E/aaiYgH7Xe5r0Oartg/+KnlS0tnjqkA4xw6aJY/l4+knrcY?=
 =?us-ascii?Q?a6fwi1Le0pl5KLBvzjPea5SX1LPnkxeEkcOLAVGW/wk90yC5EKey79x0sUhz?=
 =?us-ascii?Q?4HqCMJKl1c7jBZJb3c9pKNp4Ga6sNP2BvWI6wsT8IAG6Ag4YVoKn0x+SQzzo?=
 =?us-ascii?Q?K30KzBqzsO58y+lKOfn6eghbELZb38MOT5RLdoiXxwZ0MUBCSxO8vqDHjNIT?=
 =?us-ascii?Q?FMU7NLhR43bpyYKxj0ag0tpjjge1Im0m4A67ONFOyj+kgqMfsBNKs83z6T3a?=
 =?us-ascii?Q?Dr9QDYb7zTKBL8sy0aTFQ18UBKqXqsWHXMqfjJCjo0Ybwjt0m4zA+Q4xQBAs?=
 =?us-ascii?Q?rS0UKoNpCyPMlH8dpV4OzsCVBNvQJcp3NoQ8o1lWX8V/A02G5/UxCbEzqTSQ?=
 =?us-ascii?Q?7y0S9IlNOnJs5td9CsyNmaIWjDRfS6yJxYj72nUZxtP2e+0VFZj5d5Y+IVt9?=
 =?us-ascii?Q?j2nq0pr6+hjyzq8g5pxVuIwyVPqWd35cxV2xp0oBchFGa5kPElTqarZ+owoi?=
 =?us-ascii?Q?JAjGp72CWtgBCV9SDVPIM20T+HJ82jBC6eZI/llXnMBoqYwNgVFdMme1C5DP?=
 =?us-ascii?Q?qe2z6lNmMbE8/rssTf0ko7MRsZE/I2v2Qu7gUMAZV10CRUYfaj7Zk+dQ1yTs?=
 =?us-ascii?Q?C6zA0xPWMGepk/WPL5AXi7fAbE/L0D6o627fyyAxC8CsJKmJe7qgdB3ivhn5?=
 =?us-ascii?Q?kTxdABSujVDaed3J4mt2lJpRABBP9yiLdPDZu5t9eju1XCQiZgmoGL8GpbeJ?=
 =?us-ascii?Q?ovm9ZDYHJ84CjmhCsBMU32E+Y+6aC7J7eauVqOxCfUhW9JC/ZHf2uSy0yHsW?=
 =?us-ascii?Q?2fnZk+wCpnx169XJE9qTjXR8rCh8mo9fTJ4EApcYo6k6tjwpDCHMYPbqIVkg?=
 =?us-ascii?Q?QRrsxVUirW8Gc2TmZTatw4laqSdAI5ItODEBeV4xgm7Fa7MRJ40ELoYXudvV?=
 =?us-ascii?Q?PrydSms/IAuoZpeZfjaza5mhX5sYWg6VQbjaZ8Xo5tdc5WSJByMyTBvCDOqv?=
 =?us-ascii?Q?VuAl6F5gAGWHa2DbZYVyjy3PqjGO8ZbrL44fDoiyX87qZteMw1MuU8MTh+U8?=
 =?us-ascii?Q?Pw06urc9sC1+yfpbAVYkg2ECMJX55dPCg4+AHH9soJgt5vjKumW0cVLkA8R7?=
 =?us-ascii?Q?3yZ9A3YLSvFGKtMmVxT2m7baGGFM9sh/UDfITrrXVTE4SvEzF1eO0wvqP6qX?=
 =?us-ascii?Q?BPoU9pNkR4mS48mT+jBCN0bC66cvfBJYXCbP3CGMnxla+1l/sv3hO0h5Fvpb?=
 =?us-ascii?Q?oWHTsJbYpvhZu4N/sdTWmUUiZ1lz54knW4Feqe3YGamBN1wBmQgckykE9SIt?=
 =?us-ascii?Q?WtRWqgZiEbJ35AVKYiDRAYP8bAIxM0r55VJob1abnb0GCTMJE3qnT1eztCRS?=
 =?us-ascii?Q?AX1RXMm2SdvjeLrIOGQ/c/pHuYCanALOvaNabZ73FqUMmBGu8RcYjoPP7ImI?=
 =?us-ascii?Q?DwPcf7dIf28PheqJ/s06EXWiKo3bV6sHH8BXDFKmJg4g2ndkiXyBOHkN/dPG?=
 =?us-ascii?Q?hg=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	Qo0ZvKPTdVIvbtmuZY893lnQPS5KCBBjykVY4TYIAX1rPje0wAybnEKrYToB/2E0PlL6E3sq9vdAvszlmXkmLX/n+vVGdbFpbNwkbWfNNpwJDPyzMCvvatErGM/jJAXQUK6LPAydzIPW3HnaGL49NYuShU4FFhDLcgb2U88R07F+tJI25LX2nVusrt4z5JwgP9R82W/KXvl4OvGhyZ3efx0D1Tri9/jh44JK6CUrZAbwrKJGQl9ncL3u/HXba2UOWpiWICVtxZsWvQtSAzQE9m9hUUpCHEa/zZLSfAk6oP1P7PCn7uJzUkE13HRknV6FAOgR1NLqqFZb0cq/Hp11NYpr2ohiqMO40xPmYhOMgdWT8oq0EggVHCf9K0UJ/OGtY01vS6qnQ8yrFrLwLFjGS2jn0gAVPhejKbulqqrnk9EGinHyByhrLpQ5R7YCZdMvEI70NUq9VbYqByxYwukjqbN6a7cXMuIh+Z2syaSwJ5GQEf4+nXZaaKb6mE8YyI51IF186IuaZOfzxV8+0+50opKNbfTS+p0jMLedgs6Va2SIU4ge0yULTPL8rD0ZBARRBur3NzKTren0kUCd4A+xotRHCObPp96ZwSWcwFQ+iCc=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5abee5c5-d978-44a3-0f7e-08dc7eae11d9
X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2024 00:35:27.0077
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: cRohRK8BnUrPloW9+dQBv0KC0HDz/cX246B3omJ37Co7AxiB2ZsoF5uLCGbFJ/kYDDWRoN1hDvXtjWQqP3St/sdoRDWzol2zbX/DyB2Z9OA=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6510
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16
 definitions=2024-05-27_06,2024-05-27_01,2024-05-17_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 suspectscore=0
 adultscore=0 bulkscore=0 malwarescore=0 mlxscore=0 spamscore=0
 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2405010000 definitions=main-2405280003
X-Proofpoint-ORIG-GUID: 6rVpZOB8Yv1Q1Xth72GpDGMfuBcFBY7d
X-Proofpoint-GUID: 6rVpZOB8Yv1Q1Xth72GpDGMfuBcFBY7d

From: Sean Christopherson <seanjc@google.com>

Use preempt_model_preemptible() to detect a preemptible kernel when
deciding whether or not to reschedule in order to drop a contended
spinlock or rwlock.  Because PREEMPT_DYNAMIC selects PREEMPTION, kernels
built with PREEMPT_DYNAMIC=y will yield contended locks even if the live
preemption model is "none" or "voluntary".  In short, make kernels with
dynamically selected models behave the same as kernels with statically
selected models.

Somewhat counter-intuitively, NOT yielding a lock can provide better
latency for the relevant tasks/processes.  E.g. KVM x86's mmu_lock, a
rwlock, is often contended between an invalidation event (takes mmu_lock
for write) and a vCPU servicing a guest page fault (takes mmu_lock for
read).  For _some_ setups, letting the invalidation task complete even
if there is mmu_lock contention provides lower latency for *all* tasks,
i.e. the invalidation completes sooner *and* the vCPU services the guest
page fault sooner.

But even KVM's mmu_lock behavior isn't uniform, e.g. the "best" behavior
can vary depending on the host VMM, the guest workload, the number of
vCPUs, the number of pCPUs in the host, why there is lock contention, etc.

In other words, simply deleting the CONFIG_PREEMPTION guard (or doing the
opposite and removing contention yielding entirely) needs to come with a
big pile of data proving that changing the status quo is a net positive.

Opportunistically document this side effect of preempt=full, as yielding
contended spinlocks can have significant, user-visible impact.

Fixes: c597bfddc9e9 ("sched: Provide Kconfig support for default dynamic preempt mode")
Link: https://lore.kernel.org/kvm/ef81ff36-64bb-4cfe-ae9b-e3acf47bff24@proxmox.com
Cc: Valentin Schneider <valentin.schneider@arm.com>
Cc: "Peter Zijlstra (Intel)" <peterz@infradead.org>
Cc: Marco Elver <elver@google.com>
Cc: Frederic Weisbecker <frederic@kernel.org>
Cc: David Matlack <dmatlack@google.com>
Cc: Friedrich Weber <f.weber@proxmox.com>
Cc: Ankur Arora <ankur.a.arora@oracle.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Ankur Arora <ankur.a.arora@oracle.com>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>
---
 Documentation/admin-guide/kernel-parameters.txt |  4 +++-
 include/linux/spinlock.h                        | 14 ++++++--------
 2 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 396137ee018d..2d693300ab57 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -4722,7 +4722,9 @@
 			none - Limited to cond_resched() calls
 			voluntary - Limited to cond_resched() and might_sleep() calls
 			full - Any section that isn't explicitly preempt disabled
-			       can be preempted anytime.
+			       can be preempted anytime.  Tasks will also yield
+			       contended spinlocks (if the critical section isn't
+			       explicitly preempt disabled beyond the lock itself).
 
 	print-fatal-signals=
 			[KNL] debug: print fatal signals
diff --git a/include/linux/spinlock.h b/include/linux/spinlock.h
index 3fcd20de6ca8..63dd8cf3c3c2 100644
--- a/include/linux/spinlock.h
+++ b/include/linux/spinlock.h
@@ -462,11 +462,10 @@ static __always_inline int spin_is_contended(spinlock_t *lock)
  */
 static inline int spin_needbreak(spinlock_t *lock)
 {
-#ifdef CONFIG_PREEMPTION
+	if (!preempt_model_preemptible())
+		return 0;
+
 	return spin_is_contended(lock);
-#else
-	return 0;
-#endif
 }
 
 /*
@@ -479,11 +478,10 @@ static inline int spin_needbreak(spinlock_t *lock)
  */
 static inline int rwlock_needbreak(rwlock_t *lock)
 {
-#ifdef CONFIG_PREEMPTION
+	if (!preempt_model_preemptible())
+		return 0;
+
 	return rwlock_is_contended(lock);
-#else
-	return 0;
-#endif
 }
 
 /*
-- 
2.31.1