Received-SPF: pass (google.com: domain of linux-kernel+bounces-159855-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Received-SPF: Pass (protection.outlook.com: domain of arm.com designates
 63.35.35.123 as permitted sender) receiver=protection.outlook.com;
 client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com;
 pr=C
Authentication-Results-Original: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=arm.com;
Message-ID: <171227c2-8c19-45a0-b648-600a5e8f1150@arm.com>
Date: Fri, 26 Apr 2024 11:16:23 +0100
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Content-Language: en-US
To: Peter Zijlstra <peterz@infradead.org>
Cc: mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org,
 dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com,
 mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com,
 linux-kernel@vger.kernel.org, kprateek.nayak@amd.com,
 wuyun.abel@bytedance.com, tglx@linutronix.de, efault@gmx.de, nd
 <nd@arm.com>, John Stultz <jstultz@google.com>, Hongyan.Xia2@arm.com
References: <20240405102754.435410987@infradead.org>
 <20240405110010.631664251@infradead.org>
 <3888d7c8-660e-479c-8c10-8295204e5f36@arm.com>
 <1461277e-af68-41e7-947c-9178b55810b1@arm.com>
 <20240425104220.GE21980@noisy.programming.kicks-ass.net>
 <20240425114949.GH12673@noisy.programming.kicks-ass.net>
 <20240426093241.GI12673@noisy.programming.kicks-ass.net>
From: Luis Machado <luis.machado@arm.com>
In-Reply-To: <20240426093241.GI12673@noisy.programming.kicks-ass.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
NoDisclaimer: true
X-Microsoft-Antispam-Message-Info-Original:
 =?utf-8?B?QUpGVWROVkRsSU9XWG8wWE5taURzNGdURktiSnRRRTV3V2x1OURXTHhhUWdT?=
 =?utf-8?B?TXBmaTZYTXlrNGtXQUlPcGhLQzc5YSsyR2R3dEo5bC9iNW5ZSFhGU2lRZ3Fk?=
 =?utf-8?B?b2lHV3NrRmdPbnJPalR2TzRzaDArcE8wbk5zMXErcEs1V2NoSk80MWE0aWVF?=
 =?utf-8?B?allwcTVYUnVtbFBUOE0rWjRhQTIzUFhaOXNnOWc4ODlTMjRRR2ppRzhFK3BZ?=
 =?utf-8?B?dmRqckw4eU1tVjVubUNtakRtaFlWeFg4SWlIUVQwaFcrNzZFbWFkVUEvcEs5?=
 =?utf-8?B?QlFvQ1lza1NYRmV2TEZKa0ZaMHVzaFZjZ1MvNVhETjViSUxoNlZxazNBcDc5?=
 =?utf-8?B?TzRyZE44K244RCt3d3dqUE5RVjROUTRUWitxV1BYcS8rMGhSWkZxVTRwNG9N?=
 =?utf-8?B?MFJ6MjhEVkpsK0xwbXVnekRQUSt0UGpBSE9uelJmWHlGRHo2SE9VQmFxWWVT?=
 =?utf-8?B?UkVhRXRORGo4UksyN2ZjZVpYUmRxVWI5TUJxTFVyQ1AvckRKcGZrbGUyZ3VN?=
 =?utf-8?B?Nkp3Z0JIejdNcHIwWkI0Uno1TWFHYXFCN3RMRTFPRG1GWFRld3dMaEl4ZHF0?=
 =?utf-8?B?aDZoWFNjK09aTEsyTGJra2k4S1A3MTlqQysyWWMzY2VrV2dhUUxWSDhwZjUw?=
 =?utf-8?B?SWlRalN0UnkvWDFZRHBrVUg5cXZwUHRKUDBqOWQ2VkRvY3VzZ3pHUFEydzdI?=
 =?utf-8?B?MmFvby9taHRQNnZOUm1saDBZZzVmVmxWVk1kMHh0V0FudjF2cnZpYmRuRHJQ?=
 =?utf-8?B?YkRZclJ2RjV6bHJEbmhtZ0ExUmMrVWF5b1B1L1ZxcWs0OE16RUtjWGIwS2F2?=
 =?utf-8?B?ME4vaW1SU0t4RXlhR2duOFB6Q0hjQWdwYnZYYzBHRCt6VDY2R0VycjN1WWpE?=
 =?utf-8?B?QXJ1UkowaE5veCtZUVI2SFVCR2huNWxld0FZdk9MQnZuWkgrNHJYRk9LWDl4?=
 =?utf-8?B?MmVadEtZZjJSTzIwdEJzMjBDaXlSbEtMWVRNbDRQaXlhYmdyT0dwajBWZVF2?=
 =?utf-8?B?MHBJcVBKMDYwRUh3NGJDdGpwRE53WXFpckFkeVFVTXUzbjBmdVZXTllHbDBF?=
 =?utf-8?B?VjBDMjhGWEJKOEhMaWpzYncyMzRTTmJpb3c4cGZNRFhFTDhQV0ZaaFdqVDdQ?=
 =?utf-8?B?RGFsVlJWY1VjdG9CTlNwWjZ6TFh6bDlNOVd3QXdEeUhodHJaWjBtMDRDUjk3?=
 =?utf-8?B?d0VBSHF4cmxNOXZQb2NZb0Z2OGdWNXJGalVUTW5oRzBFekRBdGVFcmVWajBW?=
 =?utf-8?B?VC90cmNtN2l6UlM4MVlTT3JPS0t6OTNZeTlSbVZkUHE0SU56MVhPZlV0Nllr?=
 =?utf-8?B?VFFRbm9ZZjlDVUUxS0tPUEFPZXE4WXVNRFI1TXJPeG5OVWpHYWZaMFNoMnRq?=
 =?utf-8?B?YWZ0Y21ZeURqK3JvK1dnOHpId1RRWm9vbWd1MThKNFFoaDFTOWxibHZMVllx?=
 =?utf-8?B?amRIWjBrRGZHRC9zR2k0V3FSQlRXb2NkVlRWaDFHbitxaEpYWnBQS3JBSEhM?=
 =?utf-8?B?bzFtM2ttMVFZaktKTFFzc0doVnFTd1VDQkp3c2JuRzgwdmNzNXNZQzg1cXpy?=
 =?utf-8?B?UFRsbW9iZk9LdklGY2tHd3BkN1hwd295WWJBVWdaWmI3aFZVemtrQjRLWURw?=
 =?utf-8?B?c2dETHZGYnlPWHAyYU16Nk5SWlFLeG9ZVU53ZUFXWlp2bUxqNG9xOGFJaXla?=
 =?utf-8?B?ZVNQMFg5b1dPZy9nZVFEVVBTVkN0Z3RRZDZXY3JVSjc0d0QyQTVGaDBnPT0=?=
X-Forefront-Antispam-Report-Untrusted:
 CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR08MB3919.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(1800799015)(7416005)(366007);DIR:OUT;SFP:1101;
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR08MB9193
Original-Authentication-Results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=arm.com;
X-EOPAttributedMessage: 0
X-MS-Exchange-Transport-CrossTenantHeadersStripped:
 DU6PEPF0000B61D.eurprd02.prod.outlook.com
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id-Prvs:
	28dd7764-d033-4f23-0673-08dc65d9eda4
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	=?utf-8?B?YVRyK0RLWmx2eWlFSVoxSUg5WEtnK3hBUVh5aVdsT01CUEdYRUtuektUdXFK?=
 =?utf-8?B?SW5uK2JFNjJ4Q2FFMkQwUmZ5RDhoV1U5WEdld1lHbkEwRldMNDFCYlh2RFUx?=
 =?utf-8?B?VDdqdG5OTk95bjlZNTVXUmpVMjlIdUQ1REJEVHVOeWhNc3FIcnZ0eTVadFBU?=
 =?utf-8?B?bDRuVUxoWC8yQUlHNk8zWkdhR050Um9ORlVNR0NkWHE5d0JUT25pcGxjcW9n?=
 =?utf-8?B?cDlJcGppaW9OcVlWYUJVbUFPOFhuL0EwQ2Q2RWE0QTE3L3VkT0cyV0R2MUow?=
 =?utf-8?B?cjBvNFlPNWk0T1k2MUJZcGhhT29SNGZGVGF5OGtPQjZ1RmtCS2Q4d1pRSTFn?=
 =?utf-8?B?TmZvUmdqbHhDTVE4L2tqcWUxMDlOQzdZVEtvdTk0ZVNLU25Zako0K3FjUzFR?=
 =?utf-8?B?ajduTnQxdGdZZ2d3bUxQb2RBNzlEL0hFWWxVYVlEVHhGbG5tNXBpR2NlaGM1?=
 =?utf-8?B?UVlaS29ib0d6T0ZYSURlZUh6WmQvWm5uMFdRNGFRV2M0c1FqL1dCbnlCZm8w?=
 =?utf-8?B?MjBrS3ZNNUZ6WEFKcGc0QW5mS2swck1FVm5mSm5vbW1UN3NNeFZFeFNiZ0lG?=
 =?utf-8?B?ZmVmbGQ0eUcvcEpkMlIyUHhRVHJLM0EvK05GRmNiS0ZDcjJkdllFbXNmN1Vr?=
 =?utf-8?B?aUJMK1pROG1iZE85TlF2U1FUcVlXSUlacTd0MSs4RW1xZzZhMDZoR2ZlbjNn?=
 =?utf-8?B?NldXVkJIQUwrQ25YTWQrOUhnMy9CYnNXU2tOVWxybnN5Wm4rK3BCWGpDUmdQ?=
 =?utf-8?B?TEFZbUhOekF3RVZzNkd4NGJiYmxDZUt2NVhWNDdraU0xMGFwU1lPbEdRZjhD?=
 =?utf-8?B?dUdCaHBBWkR3NGd5eDd4OSs0bGxrQzVMem5wcEZKMWplRTZiMG0xUlJFL2tS?=
 =?utf-8?B?Ylk0WVN3NFc1cHBIWUdaWlRrNmRFT0wxU0RscnY0Q1NIR1BZRWpQaDR3Mngw?=
 =?utf-8?B?M3hNMXRnNHQ3c2o0VEdLZjduNnFaRzk1THVpTzVUUE1PSm5tZHZlcDNFWExr?=
 =?utf-8?B?cUhGTERmSGFYODVQUDB4c0d5SEQ2Y2o5bnVoYlpuM0Q4di8rNW9qbVZxYnpp?=
 =?utf-8?B?U0J6elNJK01DQ1pGMDVUdXY5Y0hQWWxIUWVqOENuNjJBSXk5V0M1cVdEWXUv?=
 =?utf-8?B?ZkVoNytHcmpMUnVMWjB2L0I3RE9QSEhuUnluRnlXME4xL3Q4Nkx5Ylpsa21I?=
 =?utf-8?B?OW45NEFwemtLMzF5cFRnK0VkQnRINDZHRVhxSWRyQ1U4OG1sNmdON2xqQXhy?=
 =?utf-8?B?OU9KTlJCNVRKS0E5M0ZiMmRTYnFHVnlPcW1SKzU1UzZBYk1PcDlMbmN0azBD?=
 =?utf-8?B?eU90Z1BWcDBqUnBFQU1MNlhFK0hvUmlxci9rVUxvTTNsU0VURDJXNVZ1OTA0?=
 =?utf-8?B?R1pRSmZTWU5OWTVTWEVOTDhYd0t5bmlJTW84WUxPUDI3Tkt0VkRTelUvUUxF?=
 =?utf-8?B?a2FWd1pkdXRLKzNLdVJDOGEwVUpRdmxCSVFwTXNtcENXZG9JSXRRZ0ZVcDFr?=
 =?utf-8?B?MEE0QjdrZUtNMXJxMUIzVzc0Y3FndG1NVFIvV2N2eE9SYU56bFRKWnVYR3BC?=
 =?utf-8?B?dUFUdnhFVlYwaXFKbGxTeHNwOUt1Tm5kY3NsT2IrZ1phY0h0T3RyTElDZ2tu?=
 =?utf-8?B?K05MMFFrSjVBcW5mOHVZd2lLZU5tTDZGZVNrS1BkZ1dIMDNXcWRLL0lSWU9o?=
 =?utf-8?B?UG9CYWY1K2g1UVYyKzZueU92RUlKcGw4eDV5TjBkK2hVb2dXdEVXbUE3dkEy?=
 =?utf-8?B?YU1uQUwrY0NFNnJoTmRrek5lM2ZaK01rLzgySDRMNUk0TWNOcmdBL0hEcjlN?=
 =?utf-8?B?QVBhMEdMWU5XOHBlbnJENVVodEkycWZ4Nk9ISkRudzZ3WEhVMUI4ekZkWmZ2?=
 =?utf-8?Q?EnJWXx5/Ls3gG?=
X-Forefront-Antispam-Report:
	CIP:63.35.35.123;CTRY:IE;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:64aa7808-outbound-1.mta.getcheckrecipient.com;PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com;CAT:NONE;SFS:(13230031)(36860700004)(376005)(82310400014)(1800799015);DIR:OUT;SFP:1101;
X-OriginatorOrg: arm.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Apr 2024 10:16:35.3748
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: f7693c68-7adc-4aad-1d52-08dc65d9f3e9
X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d;Ip=[63.35.35.123];Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com]
X-MS-Exchange-CrossTenant-AuthSource:
	DU6PEPF0000B61D.eurprd02.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI0PR08MB10486

On 4/26/24 10:32, Peter Zijlstra wrote:
> On Thu, Apr 25, 2024 at 01:49:49PM +0200, Peter Zijlstra wrote:
>> On Thu, Apr 25, 2024 at 12:42:20PM +0200, Peter Zijlstra wrote:
>>
>>>> I wonder if the delayed dequeue logic is having an unwanted effect on the calculation of
>>>> utilization/load of the runqueue and, as a consequence, we're scheduling things to run on
>>>> higher OPP's in the big cores, leading to poor decisions for energy efficiency.
>>>
>>> Notably util_est_update() gets delayed. Given we don't actually do an
>>> enqueue when a delayed task gets woken, it didn't seem to make sense to
>>> update that sooner.
>>
>> The PELT runnable values will be inflated because of delayed dequeue.
>> cpu_util() uses those in the @boost case, and as such this can indeed
>> affect things.
>>
>> This can also slightly affect the cgroup case, but since the delay goes
>> away as contention goes away, and the cgroup case must already assume
>> worst case overlap, this seems limited.
>>
>> /me goes ponder things moar.
> 
> First order approximation of a fix would be something like the totally
> untested below I suppose...


Thanks Peter. Let me give it a try and I'll report back.

> 
> ---
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index cfd1fd188d29..f3f70b5adca0 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -5391,6 +5391,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
>  			if (cfs_rq->next == se)
>  				cfs_rq->next = NULL;
>  			se->sched_delayed = 1;
> +			update_load_avg(cfs_rq, se, 0);
>  			return false;
>  		}
>  	}
> @@ -6817,6 +6818,7 @@ requeue_delayed_entity(struct sched_entity *se)
>  	}
>  
>  	se->sched_delayed = 0;
> +	update_load_avg(qcfs_rq, se, 0);
>  }
>  
>  /*
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index d07a3b98f1fb..d16529613123 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -810,6 +810,9 @@ static inline void se_update_runnable(struct sched_entity *se)
>  
>  static inline long se_runnable(struct sched_entity *se)
>  {
> +	if (se->sched_delayed)
> +		return false;
> +
>  	if (entity_is_task(se))
>  		return !!se->on_rq;
>  	else
> @@ -823,6 +826,9 @@ static inline void se_update_runnable(struct sched_entity *se) {}
>  
>  static inline long se_runnable(struct sched_entity *se)
>  {
> +	if (se->sched_delayed)
> +		return false;
> +
>  	return !!se->on_rq;
>  }
>  #endif