Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33;
Date:   Thu, 28 Sep 2023 16:23:06 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Ingo Molnar <mingo@kernel.org>
CC:     Peter Zijlstra <peterz@infradead.org>,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        Ingo Molnar <mingo@redhat.com>,
        "Vincent Guittot" <vincent.guittot@linaro.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        Tim Chen <tim.c.chen@intel.com>, Aaron Lu <aaron.lu@intel.com>,
        "Dietmar Eggemann" <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        "Daniel Bristot de Oliveira" <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        K Prateek Nayak <kprateek.nayak@amd.com>,
        "Gautham R . Shenoy" <gautham.shenoy@amd.com>,
        <linux-kernel@vger.kernel.org>, Chen Yu <yu.chen.surf@gmail.com>
Subject: Re: [PATCH 0/2] Introduce SIS_CACHE to choose previous CPU during
 task wakeup
Message-ID: <ZRU36iTsA7IPzhL+@chenyu5-mobl2.ccr.corp.intel.com>
References: <cover.1695704179.git.yu.c.chen@intel.com>
 <ZRPhC7b6OT1uFtNE@gmail.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <ZRPhC7b6OT1uFtNE@gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?UUwobaGtXDXxu+WHvHg6bt7ask67Z6izenI/aR28+LiKz1E1KbCgoJLwH50a?=
 =?us-ascii?Q?lMnftIWUB9g/K192cNNpVOIGjdmheqk223PVl0THzDpi3ef7DmU/aYiDsq+S?=
 =?us-ascii?Q?afuXR9Oo1iJF8YgfMBU/SOWA5Ts8ZR+QYzUcO8ToRRk0DrMFfdSJ41yzsgYj?=
 =?us-ascii?Q?JPNDxOewW7axt/aaSXlWSiMSMOC7cBxHnpt6GRIVKmQV1e2iqV6CAfiBUWOV?=
 =?us-ascii?Q?UDi6WuWRnBc8CVsQDCpdpe8Ipe2AWLTiu4rpGcA/yVoSHPZvfEihbNrkj/HN?=
 =?us-ascii?Q?DNpeFmiN/Q6O2QPl9qgHFrKF/T+Po534BAT6yGVK7d+QPX30iBGzFvJrGc+E?=
 =?us-ascii?Q?53xot/NJyxdMh6wY2k9IvkuWy5vHWlGk+J2MawG13AO/mSt1r+utUL1fMM+z?=
 =?us-ascii?Q?CYSvZmPlX2ZuJyMcKMOjr+m0tliN6lcHn7CgifTdM0dyRPbprT1ubXJQphqy?=
 =?us-ascii?Q?vkrm2GUUsquZtPFSdsSd5+h6o4X+3y6jLQJgdTs73hObQ93yqmMttikacJZB?=
 =?us-ascii?Q?owOrGr2prsPY1bGa+0scApwqnam/f2UzFkNI7BDUHQ0Hy6zxSABZxks6Sjew?=
 =?us-ascii?Q?OI3kffYYVCZIHS6PDYb3HngYV/8jnnZILNBxqYVnAjTqwM7N+gRKPGZLpAjM?=
 =?us-ascii?Q?l0GYq03SO5bQ14wvog/oY5cU29zIIgKJUUm6k1NjIqrEyKTduQ1MWjoGPZbT?=
 =?us-ascii?Q?d2sH8oL1iiZg6b5BierrXypM2mbf3wTPYgrykn4ocFslfGobfm0KOYkph6Zw?=
 =?us-ascii?Q?J5ffs84y6l+V60AtH/aiOGs4sS7Bt+aZtt47gx5F7pkWCEZcqcpZSeCPFXqf?=
 =?us-ascii?Q?KmrtWS1cm+J+8sJNuX+aGqRLG8mnVvuzHrO+0qxrZBO3U8+r32ytSAcYldiI?=
 =?us-ascii?Q?uTD3RvPbhLsIVHLVaOnFGFvvHSQrQVdM9S0+XzTo91CiX+C57jR3v5K4WCgr?=
 =?us-ascii?Q?nvVDEO3eOacIdUp+OL7v+NV8Nu+zWd3JHJ1yee7Ijc9Q7cblZ6owSNyhdc2w?=
 =?us-ascii?Q?DMoU9+CurHIg25C+FErr1C2E3DIphZTMuBEHWKk+gzYfDETTF+VdZdVzs/Zm?=
 =?us-ascii?Q?Z3Mx4BSguznVsE69XkFfDlupaKCIOAureuh0OJ14lsyYVX7OCOpAcuyBlH/S?=
 =?us-ascii?Q?uxzEqtjaIoKRncgDDrwEHUviCeM2/WJmRNZKqWyQZnsRMyLlB9xIQv4B5Ojq?=
 =?us-ascii?Q?y3fbPygMEAtJJmj8YxNxVMzLQONSNvmvvWEZHVbHsps3JMCM5MCKDZxDJzcC?=
 =?us-ascii?Q?fEuDA1EizH2tv65CYW7XcbBP8YU9JQwr4v18PBxMdRAXt7I8LKUh0fXi1FcZ?=
 =?us-ascii?Q?S7Tw6j7lKv6lAEpFGCixatmBUecqmW6CFORyHNfTiJcnHyBh/OlrAQk47CTU?=
 =?us-ascii?Q?FjNP1n1JmEGAsvtNRAxMdonIJdbMn3UHO7z1urSpIfYI5a9fVjsBVls0vkLv?=
 =?us-ascii?Q?0QbhiXJ+y+5c2XDdu3UWdwViyTnFiNvgCIGHn9m4n+ZMorER1QE6DfWrx8VN?=
 =?us-ascii?Q?MRutTTUUTWyzjxxFegPjvnpCTLnS3aCaXpgOnNbOcYXYPDBwdjKt+1PS0NBv?=
 =?us-ascii?Q?LOyYI0qPWqYE7Kbnpt1bySq29a7E+Y7XldboBYfy?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 3d7a8194-ce4c-4bed-08d9-08dbbffc2e16
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Sep 2023 08:23:23.0765
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: em+GoWr49tezXuyH8a3J5F79mv9KbLWuwFBWiFrFFOrgT00k9XOHsSf1ebTVnUTQmQKwwWsf4lFXBsKca5z0KA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB7601
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Thu, 28 Sep 2023 01:23:53 -0700 (PDT)

Hi Ingo,

On 2023-09-27 at 10:00:11 +0200, Ingo Molnar wrote:
> 
> * Chen Yu <yu.c.chen@intel.com> wrote:
> 
> > When task p is woken up, the scheduler leverages select_idle_sibling()
> > to find an idle CPU for it. p's previous CPU is usually a preference
> > because it can improve cache locality. However in many cases, the
> > previous CPU has already been taken by other wakees, thus p has to
> > find another idle CPU.
> > 
> > Inhibit the task migration while keeping the work conservation of
> > scheduler could benefit many workloads. Inspired by Mathieu's
> > proposal to limit the task migration ratio[1], this patch considers
> > the task average sleep duration. If the task is a short sleeping one,
> > then tag its previous CPU as cache hot for a short while. During this
> > reservation period, other wakees are not allowed to pick this idle CPU
> > until a timeout. Later if the task is woken up again, it can find its
> > previous CPU still idle, and choose it in select_idle_sibling().
> 
> Yeah, so I'm not convinced about this at this stage.
> 
> By allowing a task to basically hog a CPU after it has gone idle already,
> however briefly, we reduce resource utilization efficiency for the sake
> of singular benchmark workloads.
>

Currently in the code we do not really reserve the idle CPU or force it
to be idle. We just give other wakee a search sequence suggestion to find
the idle CPU. If all idle CPUs are in reserved state, the first reserved idle
CPU will be picked up rather than left it in idle. This can fully utilize the
idle CPU resource. The main impact is the wakeup latency if I understand
correctly. Let me run the latest schbench and monitor these latency statistics
in detail.
 
> In a mixed environment the cost of leaving CPUs idle longer than necessary
> will show up - and none of these benchmarks show that kind of side effect
> and indirect overhead.
> 
> This feature would be a lot more convincing if it tried to measure overhead
> in the pathological case, not the case it's been written for.
>

Thanks for the suggestion, Ingo. Yes, we should launch more tests to evaluate this
proposal. As Tim mentioned, we have previously tested it using OLTP benchmark
as described in PATCH [2/2]. I'm thinking of running more benchmarks to get
a wider understanding of how this change would impact them, both positive and
negative part.

thanks,
Chenyu