Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 27 Mar 2023 18:42:35 +0800
From:   Aaron Lu <aaron.lu@intel.com>
To:     Peter Zijlstra <peterz@infradead.org>
CC:     Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: rq lock contention due to commit af7f588d8f73
Message-ID: <20230327104235.GB571475@ziqianlu-desk2>
References: <20230327080502.GA570847@ziqianlu-desk2>
 <20230327090951.GA4253@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230327090951.GA4253@hirez.programming.kicks-ass.net>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?1tgV1CMicuEV9drv3me2MLduG3/LchXYAlhzWW9dd/xsVSZaLygbllDPeR3e?=
 =?us-ascii?Q?PIReWi7QSxXrY6fXuexcEkhwrgzK36nDL1pwOFy7gSYm02cH6tZSx420WXIw?=
 =?us-ascii?Q?jhcHpL8FBDvyq2mo0HKzWH8sAvNRSasmvj65eX8DBSMPyQoWOwfH20uWA4mG?=
 =?us-ascii?Q?ue4Dfiyz1zUkRLIqwDye+gR0tYuCVlp/Pa9F7O5qTgBr9UgJK/xu6e1XIxh0?=
 =?us-ascii?Q?5jp7ByNlWEDbkcc+ltXbpe+00Gs4edhTfcM5e4bQb0/mYfftHePd+vAa7XTn?=
 =?us-ascii?Q?2kXPxp5Z4tFWb92AzSoHhNMxjdHHbiTOiB9/0BFft+xWPn5xT3HPbtI53OK8?=
 =?us-ascii?Q?5TJpErOwW4NfzjqlpL+5ynr6+iHDNpCz9fJ9bD0cAUPTnj5E9hsicf1B5dtk?=
 =?us-ascii?Q?DI4CxhQYKWLG2NMafqXhK1N89izZyDO6pXpdDP2+P3IjSqtcXQhfA7BMbHHC?=
 =?us-ascii?Q?DV7tiKwjXTwW3iAt+3w4fEpqquA2rtPGYqR0cM4SjSmEX72WgKoFqxOOXuo9?=
 =?us-ascii?Q?ilGbrvo2ndceFSswuD4j+/Qi6CPtWUk6RsA8hXVR0k/nKDXb+0CS/Bt89zmj?=
 =?us-ascii?Q?bT+MojRFRqvoCJbrYbXkaaFIjx9tfLbvF1RyaGv7T2qHyyGgRrrEblYLjTSc?=
 =?us-ascii?Q?io/hexHc4gNLKuRrKw5G0py0U/CQMg1NIegDyRT4mdZ6mecr5WeXKvWRaoSL?=
 =?us-ascii?Q?vJIS3o7FC/Aok/ovTQQFduDL/At4m02qng+fxesN5THn7/4Oy0Mu6Wu2bHbe?=
 =?us-ascii?Q?o8nqXYTWlUtfZ1U8pQl0WvvisVAoIy9m0oTisE3uLKlerQkpsXVxLQOe7pGi?=
 =?us-ascii?Q?jbPQOMGSZOfDHaLl1BU8hvs4DAY9HObpIiZnZckLiekfPhno3NMym0xr6F1s?=
 =?us-ascii?Q?mAOSqMvtmC7I/J9JxdAsCk5TyerS4PrJD8SoQbgIqce+BzSrE6d+630ZbBV8?=
 =?us-ascii?Q?v5mxojh7GRHP63oNciAujSOlggXe6Xa+nPBnWZge1iBxyZaN0nL1/xCrF/9M?=
 =?us-ascii?Q?IQs6ZaRrGVE4+cCDTIiX0uZTv0tu3lcumovITb6rG8GRufP5IBl0GtHS2izz?=
 =?us-ascii?Q?8ZhoCMNitvaKcGpPtfis24Ru1KsuZhz9DDdJh9RmoJvMYVK+qXkIBr/02jNe?=
 =?us-ascii?Q?GpQmkHUGFCPuM7f07gkDBKiTtw1NRdA2t2ZZgf2trQGWpOrz5alA3s4+Hgvm?=
 =?us-ascii?Q?8mVM2hINDOyNrW4jLFS6Y3s4iqP3og7Nzl2AOx6ke1mIPLnk783py916LJdv?=
 =?us-ascii?Q?/ctLgVvZjrgH9E4LRYUe9bFafjpoLStWf+Pee1RDgNiT5/FVzUvpwYmpHBzG?=
 =?us-ascii?Q?RaWQJBEH9AkUqVHpctBf59AGuJZeFdthFq6Nn0bCmXoo0Qbd6XMSiDHyaDFD?=
 =?us-ascii?Q?2JWiSw78wbEBBYKxQFRU6tBujxQWOmV/0J3jrLx5gxvJF/U2M++7UrkmARey?=
 =?us-ascii?Q?GB1N7TYULf7vNWtNAB3SJNUC4IC6WN73zipHVp/yGCW7E9OG/wheYCQ9+R4X?=
 =?us-ascii?Q?WBbtwAEbRv2jf4iPUT1hC1MppB7JqwuQqsutz18CEr3O+vqISMZl6PXBy4Tn?=
 =?us-ascii?Q?lLf1aRu1nA9VC3Vdbbys//nDY4cPWkYpwVlVeXhk?=
X-MS-Exchange-CrossTenant-Network-Message-Id: bd62b9cf-5ed5-4d25-a88e-08db2eafff6f
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2023 10:42:44.3813
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: kR4IOxx9OEchNDVclsYRhuVXPMBcr554YopHHvOhnJ47Y3TtNsUa8uZmgi7sYQO5vW4masj5nvfQ7dX9lp4uGQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB7017
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Mar 27, 2023 at 11:09:51AM +0200, Peter Zijlstra wrote:
> On Mon, Mar 27, 2023 at 04:05:02PM +0800, Aaron Lu wrote:
> > Hi Mathieu,
> > 
> > I was doing some optimization work[1] for kernel scheduler using a
> > database workload: sysbench+postgres and before I submit my work, I
> > rebased my patch on top of latest v6.3-rc kernels to see if everything
> > still works expected and then I found rq's lock became very heavily
> > contended as compared to v6.2 based kernels.
> > 
> > Using the above mentioned workload, before commit af7f588d8f73("sched:
> > Introduce per-memory-map concurrency ID"), the profile looked like:
> > 
> >      7.30%     0.71%  [kernel.vmlinux]            [k] __schedule
> >      0.03%     0.03%  [kernel.vmlinux]            [k] native_queued_spin_lock_slowpath
> > 
> > After that commit:
> > 
> >     49.01%     0.87%  [kernel.vmlinux]            [k] __schedule
> >     43.20%    43.18%  [kernel.vmlinux]            [k] native_queued_spin_lock_slowpath
> 
> Unlike what the subject says, if it is due to that commit, then it's not
> rq lock but that new cid_lock thing.
> 
> Can you frob init/Kconfig and make SCHED_MM_CID user configurable and
> disable it to confirm?

Confirmed the problem is gone after disabling it through menuconfig with
the below diff applied on top of v6.3-rc4.

> 
> (also, mathieu, when you do the below, you'll see it is in a weird spot)
> 
> diff --git a/init/Kconfig b/init/Kconfig
> index 1fb5f313d18f..f2661f73f3dd 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -1042,7 +1042,8 @@ config RT_GROUP_SCHED
>  endif #CGROUP_SCHED
>  
>  config SCHED_MM_CID
> -	def_bool y
> +	bool "RSEQ Concurrency ID"
> +	default y
>  	depends on SMP && RSEQ
>  
>  config UCLAMP_TASK_GROUP