Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 4 Nov 2022 16:00:35 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Tianchen Ding <dtcccc@linux.alibaba.com>
CC:     Ingo Molnar <mingo@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>,
        "Mel Gorman" <mgorman@suse.de>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v2] sched: Clear ttwu_pending after enqueue_task
Message-ID: <Y2TGozI0YZQ7BCxc@chenyu5-mobl1>
References: <20221101073630.2797-1-dtcccc@linux.alibaba.com>
 <20221104023601.12844-1-dtcccc@linux.alibaba.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20221104023601.12844-1-dtcccc@linux.alibaba.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?CmYgqu5AL6DBH2yRjAs8P5aIz6kAn/6XIX4CDEAXJTtK22tt36KnBkwqqCME?=
 =?us-ascii?Q?AmXze2EYrqM9nyG8RUC2GoRmmSZ/Pi6MfiLLHyVq3ulVEbyokwsjvCTQ+7O/?=
 =?us-ascii?Q?RAWqeHGMRqn9moe6MTK68+jLyGJRJE67I7GyDEITmdoYDiTZBm/lNNgAOE4w?=
 =?us-ascii?Q?fxgDHdng6BWGBp+ZFD/2J7A8DoLnEjUsSleRp+jAmiHx8KyuEybljYBOgRZ9?=
 =?us-ascii?Q?sp5WTuQVWGPGmI/F3uHcSFX1LHESRbHt8DcbwSwNBKOrIHPvdHGV1frPf9PQ?=
 =?us-ascii?Q?xJ5C3ouRphvIbuCjD8a3ztK2554Ai2eYCCQ1JmN+uHDAGVRjhw3h9ElSE+Uc?=
 =?us-ascii?Q?OAmKhFDGjHjeqTwjZEHDZ3VBJGPPqlK9w3PG81AkrtdhieBzmEJVlah/qh6O?=
 =?us-ascii?Q?s1AulYapr/jwvaqvDDHKf4vnB0XqsRr6VSUK6hls+DK7fk4xfxYC3u0syjpN?=
 =?us-ascii?Q?W/4UC6KL/+/Zavm9RMYKgc33+ee0ZMYV9fF7Z0shGCqRbuOwqhv+wD07Zh4m?=
 =?us-ascii?Q?rNZAGJXA7gmOeBZrTDYSs0FujpM+wVUukt8k7D8+jF3fAw/WBAw2ifq20Qag?=
 =?us-ascii?Q?O0IBveg5w0XF5w90X6UQ05L6e9K26eWtvklxPJlGQqvGxcIoGfXqrV1TmaOU?=
 =?us-ascii?Q?IDpjxQlYEx5NLAyg4I6T7UelGtsmDx4ZluGvxlj0wika7o+4t+hBV4ZqgIa2?=
 =?us-ascii?Q?pOt0mFIVhITKBB7CT8PStD2aYRniOwcXlPKYRlcT1mKlTnntJ8PyrbEB2SaP?=
 =?us-ascii?Q?QugwhT0o4qGECVrEIBZ4jI5IqXcnR8J7+9FJDWxDI/WtdmCfX7WNawK5B3ER?=
 =?us-ascii?Q?p1QF/mOr0j4EBBDjQqqehSFJF4MxqNZ9G1hEr7fGN6uYESIg6GoWK6DsIZbj?=
 =?us-ascii?Q?ph5AP1NULOdslO+kqfOQTAd/U9AGBuUfAcwiXxe9jH0VKIQdqnQYCXYL5kCn?=
 =?us-ascii?Q?DlzjnG7vh5BcPGkpb9pCMKhVU53oNLczcVrxfejA/KLl3jizMNoGsN4cEEQd?=
 =?us-ascii?Q?nCbxTcvQnwp4vsxp0+3adO+ef+zhx++VkyuhUVLrdWNzzB72vup19fLzWhAG?=
 =?us-ascii?Q?7qeDYhGy5RZwLlay0YmmXEbllAvis+jpkiuazfsewgURtrsFw0Xa5fnpoPvt?=
 =?us-ascii?Q?nDccQcdZEqO4fVlaxDkSIW8e50Fl1nEGuhdoW/XxlSDXvl5Jh/4HGYYw4hGz?=
 =?us-ascii?Q?zry2cXcU9IIsaGnPouBzvvtl8xMgywlaQuxCMxNFmEzFg4IPsMO3wiMKAMj4?=
 =?us-ascii?Q?Rq3YPCeKVjjgfIrYuMz0FQMrf6089JAt6xx57zoLrOlkQskMfXCL0MstHQHO?=
 =?us-ascii?Q?yL0WQ+u1cbiHW/uY52HjTt2HMR4hjeCii4wYXRztHHW+6boUdlIZFmv4sKnH?=
 =?us-ascii?Q?CM8zYQxUlTgIBD6rTALLOtnP6NYQplnm4pfL+Hz9E5bU50wdHp8SwZa9GUei?=
 =?us-ascii?Q?3RFSdxC1+8HPP3ud+emkAi2jfVbbixZROO0mpODp7O+KPZFP85JM4EgKys88?=
 =?us-ascii?Q?OfwJD9cGfDzl6PFRo5ZLsdvoKp72bVKnnm574wN++nOZPUU3rAj6Kb8Oy6IF?=
 =?us-ascii?Q?aA5hcdMX40dMnOprZxyHEdvlAQ3FPDyK0oRIMjYd?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 3ce007e7-b6b2-47a4-bfed-08dabe3ab34b
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Nov 2022 08:00:55.3843
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: T3/NDlKZpVVqxS/fQyZmSMHxdKxKAhqOZ0np16nhkBF/AYiQobTCVDP78BpxMNbstuzuteCF8Uop0K//oxZlRg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR11MB5004
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_PASS,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 2022-11-04 at 10:36:01 +0800, Tianchen Ding wrote:
> We found a long tail latency in schbench whem m*t is close to nr_cpus.
> (e.g., "schbench -m 2 -t 16" on a machine with 32 cpus.)
> 
> This is because when the wakee cpu is idle, rq->ttwu_pending is cleared
> too early, and idle_cpu() will return true until the wakee task enqueued.
> This will mislead the waker when selecting idle cpu, and wake multiple
> worker threads on the same wakee cpu. This situation is enlarged by
> commit f3dd3f674555 ("sched: Remove the limitation of WF_ON_CPU on
> wakelist if wakee cpu is idle") because it tends to use wakelist.
> 
> Here is the result of "schbench -m 2 -t 16" on a VM with 32vcpu
> (Intel(R) Xeon(R) Platinum 8369B).
> 
> Latency percentiles (usec):
>                 base      base+revert_f3dd3f674555   base+this_patch
> 50.0000th:         9                            13                 9
> 75.0000th:        12                            19                12
> 90.0000th:        15                            22                15
> 95.0000th:        18                            24                17
> *99.0000th:       27                            31                24
> 99.5000th:      3364                            33                27
> 99.9000th:     12560                            36                30
> 
> We also tested on unixbench and hackbench, and saw no performance
> change.
> 
> Signed-off-by: Tianchen Ding <dtcccc@linux.alibaba.com>
> ---
> v2:
> Update commit log about other benchmarks.
> Add comment in code.
> Move the code before rq_unlock. This can make ttwu_pending updated a bit
> earlier than v1 so that it can reflect the real condition more timely,
> maybe.
> 
> v1: https://lore.kernel.org/all/20221101073630.2797-1-dtcccc@linux.alibaba.com/
> ---
>  kernel/sched/core.c | 18 +++++++++++-------
>  1 file changed, 11 insertions(+), 7 deletions(-)
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 87c9cdf37a26..7a04b5565389 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -3739,13 +3739,6 @@ void sched_ttwu_pending(void *arg)
>  	if (!llist)
>  		return;
>  
> -	/*
> -	 * rq::ttwu_pending racy indication of out-standing wakeups.
> -	 * Races such that false-negatives are possible, since they
> -	 * are shorter lived that false-positives would be.
> -	 */
> -	WRITE_ONCE(rq->ttwu_pending, 0);
> -
>  	rq_lock_irqsave(rq, &rf);
>  	update_rq_clock(rq);
>  
> @@ -3759,6 +3752,17 @@ void sched_ttwu_pending(void *arg)
>  		ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf);
>  	}
>  
> +	/*
> +	 * Must be after enqueueing at least once task such that
> +	 * idle_cpu() does not observe a false-negative -- if it does,
> +	 * it is possible for select_idle_siblings() to stack a number
> +	 * of tasks on this CPU during that window.
> +	 *
> +	 * It is ok to clear ttwu_pending when another task pending.
> +	 * We will receive IPI after local irq enabled and then enqueue it.
> +	 * Since now nr_running > 0, idle_cpu() will always get correct result.
> +	 */
> +	WRITE_ONCE(rq->ttwu_pending, 0);
>  	rq_unlock_irqrestore(rq, &rf);
>  }
>
Reviewed-by: Chen Yu <yu.c.chen@intel.com>

thanks,
Chenyu