Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 27 Mar 2023 18:14:01 +0800
From:   Aaron Lu <aaron.lu@intel.com>
To:     Peter Zijlstra <peterz@infradead.org>
CC:     Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: rq lock contention due to commit af7f588d8f73
Message-ID: <20230327101401.GA571475@ziqianlu-desk2>
References: <20230327080502.GA570847@ziqianlu-desk2>
 <20230327090951.GA4253@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230327090951.GA4253@hirez.programming.kicks-ass.net>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?UKqlghg17UXe592891QQhCj56AfqCMr+c8b4Jxi3xJHg8hctNUQLXqnDqFXB?=
 =?us-ascii?Q?dg6upOBwUCQuruRxIrfwy0FEX9cHaMOc1j67at81C4be2zRDCT56X8pgJ20D?=
 =?us-ascii?Q?AH3wCXURXVA47aP6ycPLstOsEnZEgHE5FYjUBwJHuLxHWzD3xrC56lumWYp6?=
 =?us-ascii?Q?FzvByCGrws0RfEaH7GNIRrkjArklaWQEJOibKUI5Pma6CvY3Ynr0/LrfRGDs?=
 =?us-ascii?Q?SKXqblWb44cOTHm7b2CdOyPvMs9nW4ZCRodGDBtijaPbKnNWC6KJ/3Ov7iiB?=
 =?us-ascii?Q?CTmcqUznHEWuFi8aWhJeyEFzD58zBqnH6d7Gklc2Y51HhpFYmsQcxil3+419?=
 =?us-ascii?Q?V5wzOveCktNGOfEogfia5VFWBKkMKL3WlgPMCUngdQq4K+oVgz8ZNx5vM8vo?=
 =?us-ascii?Q?w5o5K5g471qgdUOhQxFV5BNgs+tlj3RfOh5ZScfGlfCWnhc4mCUAwFxVWHwC?=
 =?us-ascii?Q?vya5shkrI/Jad5DrIX3XlXN/I59T+LQtZ5UzEnXtLoClFsNhV3zKAWAu4RaY?=
 =?us-ascii?Q?XtryI0dfxzk0EkxDz9etqUaoCaNUuE1c/kJ9iqVuVIIX0ElEp2Rg1u8pUoqx?=
 =?us-ascii?Q?u9/zNjBclRZR52VVc4sj1EyvmazOpTfoCHXZXb551olYBKe5Eqm8UUZulNDB?=
 =?us-ascii?Q?R6vcdheBSFXMirO7MB8cTtv1vNaE3nnrc96iAEEsQRsxLcVt4w4V17KbrXTe?=
 =?us-ascii?Q?OPnKDVVSE4U79iG3H08okLb/gJiLTUhgqh5J5WtgZtjmMkbL8qYvOgna+A//?=
 =?us-ascii?Q?6TY7lkg616xxVeXyhSGKGq2CRnsACFZrAO1idWbs4lLGGHqdvM1q/IjypPC1?=
 =?us-ascii?Q?C0CwmyfJMFwDB37C7fXmc6No5AXdjlpoTvUBtHX9+eUjqvIn8nh/L6FGDNZM?=
 =?us-ascii?Q?obRpihn3BTsA2ZsRdZf3f9qZYAPpQ1Th4ZZGm07M+uC4IU/sHzucIEKKK6oe?=
 =?us-ascii?Q?nVtt0+H+r7jYum5k77rFYWNvgrJPsl0e0txQhPOIpBC81Jhm1LFxOuQgKPqf?=
 =?us-ascii?Q?C6YwADL72npCEJWWnkjkl//8YRtn/btcl7856gBTr6jpbtkuz14toQ775wG9?=
 =?us-ascii?Q?44Tv2XOQPzfhBUOto6UqqCnwcWrrJj6Nn5OgTT5SJw9uoTqZ7x/2C8cK/ZwW?=
 =?us-ascii?Q?cC7bHYbLuX70llVtTeNyNniz39wC9phwV1jgn71rldeTLAVMtGHGPNoB89ij?=
 =?us-ascii?Q?FZUTs2KVXk2OxjYcKKFYgBcIqHnuWoNAkuf8fufk8+C/rHUIuHWW/lBelsH2?=
 =?us-ascii?Q?YS5r5OTNgdz4GoPy4t6V1LY9s4ZCHZlOEHiyUEnggXdwBx5DPpGko+/lrj4a?=
 =?us-ascii?Q?SJNt1N5a+belpVDOrEyuaz6qjVOSbUomoDDmd/rIAsIMxA62ld0HFbo1KaU/?=
 =?us-ascii?Q?5fjGectZuu2WEnEokuQZPrORImnEtRLZYLDiuTV2ddXussNLS4p3nuVX4z3M?=
 =?us-ascii?Q?HJWNXr7wKf0/fhKdVjvBuGFYq5Iqeb2Et35gPjFddLYYJAQlgaaZTbbfMpI6?=
 =?us-ascii?Q?Gf12SPcU7MXsrm4jqGnC34pkMtW6kks1aEkIM4Pf9dU4k3tGIk0rSfNhlAkb?=
 =?us-ascii?Q?Pn+veG8petWi6vytJK8EaZtf13F4USsZpHFj0HoL?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b485e0e4-eb47-45b1-757a-08db2eac00d3
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2023 10:14:08.7574
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 1g2PW9rhY/EEDtLXfuC/hg9NoCUWIG5TknDb/QuIw0j3qp9NwM2GLFoLNhnp/17q5RhczAiG+vKXeQ9Y1Q6L7A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5644
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE,
        SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Mar 27, 2023 at 11:09:51AM +0200, Peter Zijlstra wrote:
> On Mon, Mar 27, 2023 at 04:05:02PM +0800, Aaron Lu wrote:
> > Hi Mathieu,
> > 
> > I was doing some optimization work[1] for kernel scheduler using a
> > database workload: sysbench+postgres and before I submit my work, I
> > rebased my patch on top of latest v6.3-rc kernels to see if everything
> > still works expected and then I found rq's lock became very heavily
> > contended as compared to v6.2 based kernels.
> > 
> > Using the above mentioned workload, before commit af7f588d8f73("sched:
> > Introduce per-memory-map concurrency ID"), the profile looked like:
> > 
> >      7.30%     0.71%  [kernel.vmlinux]            [k] __schedule
> >      0.03%     0.03%  [kernel.vmlinux]            [k] native_queued_spin_lock_slowpath
> > 
> > After that commit:
> > 
> >     49.01%     0.87%  [kernel.vmlinux]            [k] __schedule
> >     43.20%    43.18%  [kernel.vmlinux]            [k] native_queued_spin_lock_slowpath
> 
> Unlike what the subject says, if it is due to that commit, then it's not
> rq lock but that new cid_lock thing.

Ah my mistake. I didn't take a closer look at the commit and seeing
__schedule path has lock contention made me think it is rq->lock.

> 
> Can you frob init/Kconfig and make SCHED_MM_CID user configurable and
> disable it to confirm?

Sure thing, compiling now and will let you know the result once done.

> 
> (also, mathieu, when you do the below, you'll see it is in a weird spot)
> 
> diff --git a/init/Kconfig b/init/Kconfig
> index 1fb5f313d18f..f2661f73f3dd 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -1042,7 +1042,8 @@ config RT_GROUP_SCHED
>  endif #CGROUP_SCHED
>  
>  config SCHED_MM_CID
> -	def_bool y
> +	bool "RSEQ Concurrency ID"
> +	default y
>  	depends on SMP && RSEQ
>  
>  config UCLAMP_TASK_GROUP