Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 27 Jul 2023 13:04:13 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
CC:     Shrikanth Hegde <sshegde@linux.vnet.ibm.com>,
        <linux-kernel@vger.kernel.org>, Ingo Molnar <mingo@redhat.com>,
        "Valentin Schneider" <vschneid@redhat.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        "Ben Segall" <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        "Daniel Bristot de Oliveira" <bristot@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        "Swapnil Sapkal" <Swapnil.Sapkal@amd.com>,
        Aaron Lu <aaron.lu@intel.com>, <x86@kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Subject: Re: [RFC PATCH 1/1] sched: Extend cpu idle state for 1ms
Message-ID: <ZMH6zVe4ezzyoNxr@chenyu5-mobl2>
References: <20230725193048.124796-1-mathieu.desnoyers@efficios.com>
 <69076f8e-191b-2e3e-d810-ea72d8ff18bb@linux.vnet.ibm.com>
 <79fa8a62-a74e-2623-9f03-1f1af85b6c07@efficios.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <79fa8a62-a74e-2623-9f03-1f1af85b6c07@efficios.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?pkURm9ylUgrJvIbxaVhbdhLSkbLKo760+NLzItZ4M/F6fNjE/svarTHvOGrF?=
 =?us-ascii?Q?7HZ1YZryttC7+gyE6RvXhN11Twz9ACVMKKGM/o7Z29Juw4CS49OnQK4YcLTd?=
 =?us-ascii?Q?05Q3xGx6syLRDP4aNSJw1XzUXp0hu/fNR4X+688zIaApmUHRzJBMTh4Bjwop?=
 =?us-ascii?Q?WmRVM9n4zuVn/mHcltf542V+mTZxsD98EdEdPTNOOXPWU77Q6O5UkrPysVYT?=
 =?us-ascii?Q?/LkVo5zGklfl6qFS6lIELknR3Q+TkC2YMG9b4lVatk/0KdRs/09lhXAFFJY7?=
 =?us-ascii?Q?UpTyuAIKRjiNUoSQFNcg9hoT6ueBaJO2X8KRUpCJMrtP08g4kU4bXD7GUoan?=
 =?us-ascii?Q?N6PFAFP/r7NVEwJoq8XKtNGweRQB/5dE70BWoXmTwDYKWvLqrOteGsMTJiHO?=
 =?us-ascii?Q?tZgP8Aj7Ysrtxwj8HSbwvwxPpNU6Dk1VQbvIC+Y8v6j9FNMRPgyyiGCGpZa4?=
 =?us-ascii?Q?AFCvV1dkOmUpT0Jtwd87ao9CEoaUxZDYceoN5BYXVOwCg87agC82sutjTPnj?=
 =?us-ascii?Q?BUWMbma6lVbA49H7zosTe97UCL4AcH6ODw1R8d+6o23tQBwQm0PUqgyz+Vkz?=
 =?us-ascii?Q?h5MeMOvy7xX97/fygZM+0vXYd3TsIEP50r2e1VlskG58GMuYjK994MTHtT1n?=
 =?us-ascii?Q?6a6hgc3r4sHNfuTzkJ6nrE4i2jBaSyKq0mGP75xzo59WIEyfhwUx50HrQHec?=
 =?us-ascii?Q?homjj9ZLm6bTCIYWT/GSqXNUkVnbQgSZTHNLQCtAxZgF/viw9T0KYknOdhgG?=
 =?us-ascii?Q?HFzBd2z0aL1lImhgWcU1kLj+qsoaMt/qTbSOQIoBtvIaw0ExHlILEX67NWjN?=
 =?us-ascii?Q?hFWXPp7Yc4MZLFPEGimvZtUWcpzp5itWdFDoR8EkJTVBZWHWpoMiGphP+Z2G?=
 =?us-ascii?Q?ksMCXyi/CFDS0ewlyopz4sqsVmCb0YvQUcDDRfGBrBjobEYan/A5myF4qMph?=
 =?us-ascii?Q?HiWklpkzsuiMAfqhdCCH0hZClsjtLPNb6Kd+PoPQHSO08XznkcJErfGdn6Eq?=
 =?us-ascii?Q?2vbyEP19pU2h++tXZFCG4xQohePCVse01sxrxkFSdP1iDVUwLAoPjLswr9jI?=
 =?us-ascii?Q?39IGWddIi5GXGUnJ6KWz7BYCGbB+lhe0bZfI1HGGnTfo2Jvk1jbgF9T49WCx?=
 =?us-ascii?Q?S+B+La1nT+Xw8ChA7/lWoIkb9Icf+T813I3j37tumGbiJz7/CMre+ROjfk91?=
 =?us-ascii?Q?PYUSmyqpsf1HJ35svIByVuJ/ADdxFNc566c9tC1g0vTKLnSUWZ9JnYDt/fnU?=
 =?us-ascii?Q?aX/2cir67Q5+xi2kgO5ftyDw6BZv+P9GNgzhVfip77QnVlIIdJFrC1GOHvIa?=
 =?us-ascii?Q?5KlTZ7yemxDq7/Om2Fag/MwJW95+ah5lZGhtFTUoZ0hrIhXWpTbF1Gny07DT?=
 =?us-ascii?Q?U1lJ3pLnFYsr26uGVgiyk6SzdzW8ErVNglN+R8lJuRjpzLlJl1zRiXZEbksl?=
 =?us-ascii?Q?3fpd3Ix0Jh9wnpYQX6u0Uzl8PPnH7zK8zHkJEhhDp4K8ARLozzRGV95SY5m7?=
 =?us-ascii?Q?ZcbkmtfcjX6Jd6ERU3Sna9lBM+jTPgOkpWUZIOTeQf6TEcvktCRDm9Lzc7qV?=
 =?us-ascii?Q?7RGvWeCehZd+sx/kp1VXcw/etigJh6tlPv+aTMYe?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 8bb6c9ab-4898-4c5d-d63b-08db8e5ef808
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jul 2023 05:04:34.6632
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: C0XC2wy0K76I5XMHfyRLP3N/j/HpAB82bvXb/0d69e49YM68mryu+MoWr0S/wO5s/DsnL6wrKMt8QEHd8H3nOw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR11MB7977
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 2023-07-26 at 10:07:30 -0400, Mathieu Desnoyers wrote:
> On 7/26/23 04:04, Shrikanth Hegde wrote:
> > 
> > 
> > On 7/26/23 1:00 AM, Mathieu Desnoyers wrote:
> > > Allow select_task_rq to consider a cpu as idle for 1ms after that cpu
> > > has exited the idle loop.
> > > 
> > > This speeds up the following hackbench workload on a 192 cores AMD EPYC
> > > 9654 96-Core Processor (over 2 sockets):
> > > 
> > > hackbench -g 32 -f 20 --threads --pipe -l 480000 -s 100
> > > 
> > > from 49s to 34s. (30% speedup)
> > > 
> > > My working hypothesis for why this helps is: queuing more than a single
> > > task on the runqueue of a cpu which just exited idle rather than
> > > spreading work over other idle cpus helps power efficiency on systems
> > > with large number of cores.
> > >

This looks interesting. And it does help power efficiency but how it could
improve throughput? Is it because of hot cache locality waking up task on
it previous running CPU(because it will be easier to be treated as idle),
or just reducing the time in select_idle_sibling()?
 
> Good point !
> 
> Can you try your benchmark replacing the if () statement above by:
> 
> +       if (sched_clock() < READ_ONCE(rq->idle_end_time) + IDLE_CPU_DELAY_NS &&
> +           READ_ONCE(rq->nr_running) <= 4)

If I understand correctly, this nr_running is to filter the case that the system
is saturated? If that is the case, maybe 
	rq->avg_idle >= sysctl_sched_migration_cost
could be checked in case there is 1 long running task and we don't want to treat this
cpu as 'idle'?

thanks,
Chenyu