Received-SPF: pass (google.com: domain of linux-kernel+bounces-158141-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Thu, 25 Apr 2024 15:41:43 +0800
From: Chen Yu <yu.c.chen@intel.com>
To: Sean Christopherson <seanjc@google.com>
CC: Jonathan Corbet <corbet@lwn.net>, Ingo Molnar <mingo@redhat.com>, "Peter
 Zijlstra" <peterz@infradead.org>, Juri Lelli <juri.lelli@redhat.com>,
	"Vincent Guittot" <vincent.guittot@linaro.org>, Will Deacon
	<will@kernel.org>, <linux-doc@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, "Valentin Schneider"
	<valentin.schneider@arm.com>, Marco Elver <elver@google.com>, Frederic
 Weisbecker <frederic@kernel.org>, David Matlack <dmatlack@google.com>,
	Friedrich Weber <f.weber@proxmox.com>, Ankur Arora
	<ankur.a.arora@oracle.com>, Thomas Gleixner <tglx@linutronix.de>
Subject: Re: [PATCH v2 2/2] sched/core: Drop spinlocks on contention iff
 kernel is preemptible
Message-ID: <ZioJN6ClnlFIQIBg@chenyu5-mobl2>
References: <20240312193911.1796717-1-seanjc@google.com>
 <20240312193911.1796717-3-seanjc@google.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240312193911.1796717-3-seanjc@google.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?tEqBeKtiC3ZCBNPRM8zfL15pWbPoelrljkmFeaPZuiK6jxsOc/qFf9kx4BK2?=
 =?us-ascii?Q?Xd4VqvxSDuMhdR4VNT/Es7cJDvSuc4XQK7+bvULrFXTnM/NVJhPC5rZWgHMP?=
 =?us-ascii?Q?SGnf4I/cKdB3rWA21bXd3hBK+qdBslFAMKCkchd5KK/zpVVWcAxyXYx+h4cf?=
 =?us-ascii?Q?AaIPpEiNJISvaRHoxrXW/DP+lor94GJCMgVu+udUfeT0xbjvScG66YeScsAd?=
 =?us-ascii?Q?F5hLhElWwh2HzGjKCroLQrohM43DmAftOV/6b6V1uwug/bYNp827jvfyJMZu?=
 =?us-ascii?Q?rO9Pt2vlcHi7842TfjQRP9oWF3LKLXSqGbePnb45ymSkYmRAPQWgT0ojuCQ6?=
 =?us-ascii?Q?5xNQb5Ff6Cu5sNT61r6o1Z0bXVxXwIpS4mY/dM8atZH2bvxAvzD58YlMDh+a?=
 =?us-ascii?Q?VxWw6hO6ePFZlGGNoxDwKUA82NW76uLz62vkla+njGZxSQXaxkYCdinNNLgh?=
 =?us-ascii?Q?eid/p2cl4ytydrgxRIcc6EFbHD8pDRM+kCoCF0R06WPPwJ26H2KHcV7byqnC?=
 =?us-ascii?Q?sWULcgpJM99AdEP5dNfb83VLnJjAvnnQbHaN1KRwZgjlJ3r8nEOwbh5wgrYJ?=
 =?us-ascii?Q?wSKlGzR2pkfrhQB+23DwdKI9exIKKXcxgSdhrP3CHfW37t9+ZgVS/Lgl+net?=
 =?us-ascii?Q?C/QlGxyVaEojp/d6+l2biThyNqtSPthQPurX5wvudFwHST6tirSna1AroO8b?=
 =?us-ascii?Q?2bsp0MwbAtQYtkTAY5zwOUiG6TTls5wpBy2z2JfWn9GEPHKMun0jri1kfKDM?=
 =?us-ascii?Q?7B+LZoG9Zug4wCfhjTfOILSygK/r2IDUDTJudmzHDCJ2wYFHOA+tzORic+Ym?=
 =?us-ascii?Q?npRXNHclOJHP1Mb3HPLreKuRk8g2uevcNohQEomeanidIV2IN5nX73rTINF6?=
 =?us-ascii?Q?QH6F7RuI0tq+MxPJRJ4sra/0pSP08R/qT7N+6vo+wFx/844FvolGmFZLbRxf?=
 =?us-ascii?Q?896L/thJh0kdhK+qwAvT/4Q6PiEHBwKjXPt+a1NG9Fnl9CCF/+Dht7pR3On6?=
 =?us-ascii?Q?AZ4OBOnjfFae+fiR34lSh6iHGLMGPU7Nyhja5biDPRVco8kEWN+H+9MokJEs?=
 =?us-ascii?Q?ekvUmH4E8OYx+W+6OVmko7vVC3fkU9vR5SxkpSELeknySclrqU8Fi4hyxG3X?=
 =?us-ascii?Q?ydZyc1ivzOnkjO+DS07hSDimUVRSjlNPRk1Se7xjoBHRN1MP6JqZHlON4CWZ?=
 =?us-ascii?Q?7VHkCOECPNhaYbzdJnfSO6sfJPcHGyxT123MM53kAuViMFSPbLk4ETX/tDOK?=
 =?us-ascii?Q?yfrr5DZaWJ1j55WprnU79CyNyMwx0HE4gs0DOqcdTFNOc443lR+1ksBWUX4i?=
 =?us-ascii?Q?A/8OO1KIGQuOMWt0bjYRq70Jvamr7uZ1bGxVRBqibTb3kSLyDjc65b8TjIVK?=
 =?us-ascii?Q?hnEsdAbcbQToDkTcN/Ujoh/DK49rMpVnyJFKn+3H3pE+IDoj/dEBLBP5z90z?=
 =?us-ascii?Q?QG0a6KzXoIO8qn6Un4tAzxjzQFEEqfU8mxgQsT5qQ7IPSUb9TvZUa1zfbH4j?=
 =?us-ascii?Q?fvOWF6nbVrUgiMy6uQavafP9OICdTWVDvsLiElkhwRvKlNI5DQ82z/jrCJ4y?=
 =?us-ascii?Q?s660GgFFirgjarMeA2mgwrJb7UkVd8NVa1fAynRR?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 0658d09c-29cc-450a-1500-08dc64fb316a
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6020.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Apr 2024 07:42:00.9702
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Ug79b28w3SinE362Jea2Y+wOPRjEFK81rUps6RndQsw6rcDJ82Gi6f1f5Zv4lEI+PHWlXFpwiEO+Xht4Aq/9gw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB7659
X-OriginatorOrg: intel.com

Hi Sean,

On 2024-03-12 at 12:39:11 -0700, Sean Christopherson wrote:
> Use preempt_model_preemptible() to detect a preemptible kernel when
> deciding whether or not to reschedule in order to drop a contended
> spinlock or rwlock.  Because PREEMPT_DYNAMIC selects PREEMPTION, kernels

It took me a while to wonder why PREEMPT_DYNAMIC selects PREEMPTION
in Kconfig, then I assume that you mean the static config is CONFIG_PREEMPTION,
but the live preemption model is "none" or "voluntary", which makes the
static check of CONFIG_PREEMPTION in spin_needbreak() and rwlock_needbreak()
invalid?

> built with PREEMPT_DYNAMIC=y will yield contended locks even if the live
> preemption model is "none" or "voluntary".


> In short, make kernels with
> dynamically selected models behave the same as kernels with statically
> selected models.
> 
> Somewhat counter-intuitively, NOT yielding a lock can provide better
> latency for the relevant tasks/processes.  E.g. KVM x86's mmu_lock, a
> rwlock, is often contended between an invalidation event (takes mmu_lock
> for write) and a vCPU servicing a guest page fault (takes mmu_lock for
> read).  For _some_ setups, letting the invalidation task complete even
> if there is mmu_lock contention provides lower latency for *all* tasks,
> i.e. the invalidation completes sooner *and* the vCPU services the guest
> page fault sooner.
> 
> But even KVM's mmu_lock behavior isn't uniform, e.g. the "best" behavior
> can vary depending on the host VMM, the guest workload, the number of
> vCPUs, the number of pCPUs in the host, why there is lock contention, etc.
> 
> In other words, simply deleting the CONFIG_PREEMPTION guard (or doing the
> opposite and removing contention yielding entirely) needs to come with a
> big pile of data proving that changing the status quo is a net positive.
> 
> Opportunistically document this side effect of preempt=full, as yielding
> contended spinlocks can have significant, user-visible impact.
> 
> Fixes: c597bfddc9e9 ("sched: Provide Kconfig support for default dynamic preempt mode")
> Link: https://lore.kernel.org/kvm/ef81ff36-64bb-4cfe-ae9b-e3acf47bff24@proxmox.com
> Cc: Valentin Schneider <valentin.schneider@arm.com>
> Cc: Peter Zijlstra (Intel) <peterz@infradead.org>
> Cc: Marco Elver <elver@google.com>
> Cc: Frederic Weisbecker <frederic@kernel.org>
> Cc: David Matlack <dmatlack@google.com>
> Cc: Friedrich Weber <f.weber@proxmox.com>
> Cc: Ankur Arora <ankur.a.arora@oracle.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>  Documentation/admin-guide/kernel-parameters.txt |  4 +++-
>  include/linux/spinlock.h                        | 14 ++++++--------
>  2 files changed, 9 insertions(+), 9 deletions(-)
> 
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 825398d66c69..fdeddb066439 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -4689,7 +4689,9 @@
>  			none - Limited to cond_resched() calls
>  			voluntary - Limited to cond_resched() and might_sleep() calls
>  			full - Any section that isn't explicitly preempt disabled
> -			       can be preempted anytime.
> +			       can be preempted anytime.  Tasks will also yield
> +			       contended spinlocks (if the critical section isn't
> +			       explicitly preempt disabled beyond the lock itself).
>  
>  	print-fatal-signals=
>  			[KNL] debug: print fatal signals
> diff --git a/include/linux/spinlock.h b/include/linux/spinlock.h
> index 3fcd20de6ca8..63dd8cf3c3c2 100644
> --- a/include/linux/spinlock.h
> +++ b/include/linux/spinlock.h
> @@ -462,11 +462,10 @@ static __always_inline int spin_is_contended(spinlock_t *lock)
>   */
>  static inline int spin_needbreak(spinlock_t *lock)
>  {
> -#ifdef CONFIG_PREEMPTION
> +	if (!preempt_model_preemptible())

The old version checks against static CONFIG_PREEMPTION, now we check
the live CONFIG_PREEMPTION and static CONFIG_PREEMPT_RT, just wonder
if the rt check is needed here?

thanks,
Chenyu