Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 1 Nov 2022 20:58:20 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Mel Gorman <mgorman@suse.de>
CC:     Andrei Vagin <avagin@google.com>, Ingo Molnar <mingo@redhat.com>,
        "Peter Zijlstra" <peterz@infradead.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        <linux-kernel@vger.kernel.org>, Andrei Vagin <avagin@gmail.com>,
        "Dietmar Eggemann" <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>
Subject: Re: [PATCH] sched: consider WF_SYNC to find idle siblings
Message-ID: <Y2EX7MlnK5rKT5aj@chenyu5-mobl1>
References: <20221027202603.670616-1-avagin@google.com>
 <20221101094157.a3gh2ko6otaa6cyw@suse.de>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20221101094157.a3gh2ko6otaa6cyw@suse.de>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?nJ7dqYqqHCc6LjMpZYDNzHmulNoePcf5Hz1/Lo2c2TpM+j/hSzv5nt+2cAT/?=
 =?us-ascii?Q?+48PdH6elw9c5gk330v9kxsPnOusriQ5fs/ZaW3wJITAKp9BGfTT8YR6k82l?=
 =?us-ascii?Q?aJV7hr9NfFhDiqMygPzLP59m8/uYYbKPcvHHMHAuhrt/KlLa8rrJu3eSuL+K?=
 =?us-ascii?Q?6lAH9j6fw0AHLcmKFCoG84qMU1hanCW8keUW9jNUXDNVW1t4AFdJ+rI8mRK7?=
 =?us-ascii?Q?bhbQezfQGYdWY6kYAo0M5KGxt9P6U/iDW4fi7pa99HQK2prIYdFp2RqNQmAX?=
 =?us-ascii?Q?AKQMRxYy4sFwLGz6mjo4tHGNdwxU560dHhP6HQ7s3NDKMX2Xv6nFSQTSSeyM?=
 =?us-ascii?Q?d2DmJ1qxhyOuWH7ia1P+O9FgGbEQYqr6kYyVyqWTpV24Eh+OICOprFUIlW9J?=
 =?us-ascii?Q?lOdDLm53UCHQAEEmK32LkVqAVkb5So3nH0IT6xv6HFcZFfnmMbnI92LlHSgz?=
 =?us-ascii?Q?oTDeFuziCtRdhLrQMT99qGWwM18wa4X5MS90okhH9rqdY8IXsayZx+zU0dBJ?=
 =?us-ascii?Q?u4OjYK0NnFcczsjgvbpxK7smExa1oGXbJVdtk6+gXGCuqWQJ3DidApdBFSm0?=
 =?us-ascii?Q?LO1zELCVY7+WG+b6JBvsDV07aB7BPN8AIhNI/eHfvImQbs+Uv+BuyPkyh4ZU?=
 =?us-ascii?Q?O5eU/UQM0AqfahO+QABwiZuFFYgYHnrmiUDjOC1Wnyc5KL2NMPjnrgKf4ows?=
 =?us-ascii?Q?VVoB+TjfUvy85hKjrY1UML1894ZbNJc5QHsEcIay76IY5UF2Jaq40JJ0XFz/?=
 =?us-ascii?Q?/zbVNrlG/JIi+aLfuU83TAmU0EpNa83K2SSHLls7GM96dnlC4r7N2QKcDk6Y?=
 =?us-ascii?Q?dEwA58bNHYkfmwZvK7zFBrGHTJMW4IaRaqtYxcJN8jkhR6qUE+wupITWop6r?=
 =?us-ascii?Q?SY7tsue72klJ4XuPF1tXuFCL1fb4OK3kx3ietmzLSNkpzf0SqfuvbwLIimTY?=
 =?us-ascii?Q?GVdZ9wdhtaNZMzlhlIub2o3qmyuTi7B0wxvSxdWIytCHXJEA2zahmXOstKmq?=
 =?us-ascii?Q?bXSWAVtD7Lmxp6VvIW61oUk3lgSDsB2n1gXrS3hG3lW/IRvICqeADJ7coZMx?=
 =?us-ascii?Q?jFDa0ztELpZOvmXtRSQUCitxqDXfFOPYseva/aubseFNkqJ0hXkswCrigAwi?=
 =?us-ascii?Q?dAtXakVtlxW9rVlT6AJN53fgLbmPnDvDOYciVI5SP6Z1VX8bAwpQVI4yq3Lw?=
 =?us-ascii?Q?skCp7qxmVCbcPxXro+J8czY1I0Yx6LHrYG8FUpb9D40kmeOwBu3SLm1lPB10?=
 =?us-ascii?Q?OXBRmJAUTxwKJ3R4As7AtQrPoZjJdNFQc6twmtTtwfCBZSxnPLfd6ob6NWjy?=
 =?us-ascii?Q?dQsaLkIVou+XE0m7xb/+qEicxJymNupaMhAazRI+5driS0mi3dUxq5fjypVh?=
 =?us-ascii?Q?Mb/PPI+GiMQ9el5fU23WaBxbaXa3Ll2pJCZzA9rSn7YGcabGUcV5+luYmIVO?=
 =?us-ascii?Q?49z4O4E8PmSFp7GT/mqV6I6tffdbsj8Xi9y8HaLfEMq94nux8D2gyRXB8IDn?=
 =?us-ascii?Q?SurAbo4PwgH0nQ/QeqJql+gSryK7KdHEwM0GP2N6REkZtl+JEAhybrgc904z?=
 =?us-ascii?Q?NRrG51bvZqT1UmbRrYHBjIYycTScZ3lj0gsjsqbs?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a74b523b-7623-44f0-08fe-08dabc08cad3
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Nov 2022 12:58:37.7759
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: MkKzuRUM5RnbPqZQwmmkb9LvuCJEtMOZJVpk6RzD6ZA++XDwANTK9o1mj0iuWr/RM3Bou2jAlyUffIpwvRC2Xw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB6193
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Mel,
On 2022-11-01 at 09:41:57 +0000, Mel Gorman wrote:
> On Thu, Oct 27, 2022 at 01:26:03PM -0700, Andrei Vagin wrote:
> > From: Andrei Vagin <avagin@gmail.com>
> > 
> > WF_SYNC means that the waker goes to sleep after wakeup, so the current
> > cpu can be considered idle if the waker is the only process that is
> > running on it.
> > 
> > The perf pipe benchmark shows that this change reduces the average time
> > per operation from 8.8 usecs/op to 3.7 usecs/op.
> > 
> > Before:
> >  $ ./tools/perf/perf bench sched pipe
> >  # Running 'sched/pipe' benchmark:
> >  # Executed 1000000 pipe operations between two processes
> > 
> >      Total time: 8.813 [sec]
> > 
> >        8.813985 usecs/op
> >          113456 ops/sec
> > 
> > After:
> >  $ ./tools/perf/perf bench sched pipe
> >  # Running 'sched/pipe' benchmark:
> >  # Executed 1000000 pipe operations between two processes
> > 
> >      Total time: 3.743 [sec]
> > 
> >        3.743971 usecs/op
> >          267096 ops/sec
> > 
> 
> The WF_SYNC hint in unreliable as the waking process does not always
> go to sleep immediately. While it's great for a benchmark like a pipe
> benchmark as the relationship is strictly synchronous, it does not work
> out as well for networking which can use WF_SYNC for wakeups but either
> multiple tasks are being woken up or the waker does not go to sleep as
> there is sufficient inbound traffic to keep it awake. There used to be
> an attempt to track how accurate WF_SYNC was, using avg_overlap I think,
> but it was ultimately removed.
avg_overlap was removed 10 years ago because of accuracy problem that
"we are missing the necessary call to update_curr()" according to
commit e12f31d3e5d3 ("sched: Remove avg_overlap"). But in current code
I think this issue described in above commit does not exist anymore because
in current code the put_prev_task() would invoke update_curr() for each
entity, then calculating the avg_overlap is always using the update-to-date
runtime? If it is true, is it applicable to bring avg_overlap back?

Some benchmarks suffer from cross-CPU wakeup which introduces rq lock
contention. Similar to this patch, I tracked the average duration of the
task and place the wakee to a CPU where only 1 short-running task is running,
which is another direction to mitigate cross-CPU wakeup[1]. Not sure if we
could deal with more accurately?

[1] https://lore.kernel.org/lkml/6b81eea9a8cafb7634f36586f1744b8d4ac49da5.1666531576.git.yu.c.chen@intel.com/

thanks,
Chenyu