Received-SPF: pass (google.com: domain of linux-kernel+bounces-202066-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Received-SPF: Pass (protection.outlook.com: domain of arm.com designates
 63.35.35.123 as permitted sender) receiver=protection.outlook.com;
 client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com;
 pr=C
Authentication-Results-Original: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=arm.com;
Message-ID: <7b26df23-4977-4fad-8721-137a23932b6e@arm.com>
Date: Wed, 5 Jun 2024 10:14:47 +0100
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
To: Peter Zijlstra <peterz@infradead.org>
Cc: mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org,
 dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com,
 mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com,
 linux-kernel@vger.kernel.org, kprateek.nayak@amd.com,
 wuyun.abel@bytedance.com, tglx@linutronix.de, efault@gmx.de,
 John Stultz <jstultz@google.com>, Hongyan.Xia2@arm.com
References: <20240425104220.GE21980@noisy.programming.kicks-ass.net>
 <20240425114949.GH12673@noisy.programming.kicks-ass.net>
 <20240426093241.GI12673@noisy.programming.kicks-ass.net>
 <c6152855-ef92-4c24-a3f5-64d4256b6789@arm.com>
 <2fba04b0-e55e-41f4-8b7a-723734fe1ad2@arm.com>
 <20240529225036.GN40213@noisy.programming.kicks-ass.net>
 <7eac0774-0f9d-487c-97b6-ab0e85f0ae3a@arm.com>
 <20240604101107.GO26599@noisy.programming.kicks-ass.net>
 <24e09046-74ee-4ebb-ac1a-bdc84568e825@arm.com>
 <20240604191220.GP40213@noisy.programming.kicks-ass.net>
 <20240605072225.GR12673@noisy.programming.kicks-ass.net>
Content-Language: en-US
From: Luis Machado <luis.machado@arm.com>
In-Reply-To: <20240605072225.GR12673@noisy.programming.kicks-ass.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
NoDisclaimer: true
X-Microsoft-Antispam-Message-Info-Original:
 =?utf-8?B?Q3A2TktFaEkvTi9KbE85QkRoYW1FbW80enV3cjNxbGJlWU45T0F4MFhrTWRm?=
 =?utf-8?B?RHdleWQyYUxsM1VvWlBNVDdwSUZwSjdNUnVUYWc2NHNkaEM4eWtVUjZwMWtp?=
 =?utf-8?B?YTQ1KytFWGd6RmR0b0ZUdUtZOHo3TUxmVm5wSHJaRStXNy9XV2FZVUFSV3VQ?=
 =?utf-8?B?WEpaVDRUZHFWZ29jdUZ5NlNuR3owc1Z1ekJHYjZ2VUFiT2ZpMDRBaVZraEIz?=
 =?utf-8?B?MFl2Y20zc3ZwNzlWM0ZxQWdWL3J3ak95V3kwSzBIdG8rUThHNC9MbmZwcVp5?=
 =?utf-8?B?c2UxMzc2K0l6aWV2OUxhdnhGMmZsY1MraHNHMHhOWk94ak4vSit3aEZHVXg4?=
 =?utf-8?B?QnAwT01KRHpBRk5RTDJ6NHgzVlExaHpINlQySGlhVlpUZE9uZ0pTK1oyZUZ1?=
 =?utf-8?B?Umo1WWI5c280a1VxUSs5OUNwVm5iSUwrSkdCWU1EcUtocGVPSUxFWUZxOWhE?=
 =?utf-8?B?R2l6aUxRSnhNZmk1UjQwdG5SbVh6TU5zZ0REN0xKeWJxVXI1eS93MHcyc21i?=
 =?utf-8?B?aVo1aVg2SlBaendyTzZQdWQzdkFEd05oT0Z6MHNWZjdqSllaUEFEWk5QUURT?=
 =?utf-8?B?c1pZQWZxell4OE5RdXRtUFBzODh6eGtQS25pTnNvd29LZzZ6MDFzSmlCQ0dE?=
 =?utf-8?B?V3dld2Q0NEhQTEJWWldacUQrK0w3TWhQYVgrRkxRa21aSkhWWGEvVUFNYjVJ?=
 =?utf-8?B?NlFBc3djT20ydXV6U1h4SVR2dmdvZkNiTlhHbEtGcEJMZ2RGczhXL1N4MWw2?=
 =?utf-8?B?OWJkTW5qUWFFUWlZRFFhMDlpQVNOb3ZiQlROOEtndkMzSEk1RFJ0dndXZlky?=
 =?utf-8?B?WEt5N1RZS29xN2laaTRWcmo1cU5uZjFnVVY4VnZ3Vno0UnFOM3gyei9HQmUw?=
 =?utf-8?B?S0JtN3BCbnl6b056WDBJcC9KUlNsRk5Mc2NkR2tpcWw1dDhXeDlmOHNYN1I2?=
 =?utf-8?B?WVBYWTZleHdaZmZLeUhVN3pUU09lcmp3c2diS3lkTlBBaTZjSllxNE1IUEw1?=
 =?utf-8?B?YjJlRm5PbEY3VXRiUWgybnBTQTBmdExCbU5FTmxpbGlpMWYxVWcxc2xjdENm?=
 =?utf-8?B?MmFuazNzRUFydFYvMmliZEdUcUJaNEJLWEsydkZmNllvZTc0R0x1TVJ3NzVn?=
 =?utf-8?B?a211N2xFeUZHY0poMmFwTWVDcUJRbVQxVVdKWmhTd0E2Qnc2aW5OWnQrSFk1?=
 =?utf-8?B?QjIzRGM2Wk56RU82bEhHRGVmeVNJVEQwSmNIcEZlcEpCcjMwMlNvTThSN245?=
 =?utf-8?B?VnpUL2JEc1RMU2tkUG9xakhGNGhmVExKdUtYK1NDOUVGOFJGMWpOUUVDMnFV?=
 =?utf-8?B?ekM4OFRzRFI3Z3RuNjRWVTVYeWVNcGdCT3dBUitzNDV1eE1FUEY4Uk9XYlVh?=
 =?utf-8?B?aHQxYndyNGE5bGtTRmVtcW9DN0FSQ2pNQjB5dTdHYjJ4UTR1QzdxcjZWUzJr?=
 =?utf-8?B?b0RUZXU1eDcrZTZMaHNMSktNSkxaQ2lrMUM5TEtCcm5TSyttdTRJSVlhdVYx?=
 =?utf-8?B?eFlQUzdOUkNqMFczNTk0OWlyRnBuOUdzQnFDZW51V01XZWFaV2lhbGo1Tnoz?=
 =?utf-8?B?b3lQYzV4eEMzMzYrTWttT29IRElEc2hPVGFHVHdObFpYRXJGRHYxNS9Tejhy?=
 =?utf-8?B?UlNmQVo1WGp1UmFiZnpxd0poSjhBK0hLMm15YWNCSi8vSXR5cTBJcGFvR0NN?=
 =?utf-8?B?WWFLQkxja0tCVHlsY2luL3VFU1o4L0dLdzJPUVRYeTN1L2ViaDYzeEVuMkdV?=
 =?utf-8?Q?MF6S06OIFsdo1Se6foJFn0w/mtkCUl5qFgeXKJO?=
X-Forefront-Antispam-Report-Untrusted:
 CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR08MB3919.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(7416005)(376005)(366007);DIR:OUT;SFP:1101;
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS2PR08MB8645
Original-Authentication-Results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=arm.com;
X-EOPAttributedMessage: 0
X-MS-Exchange-Transport-CrossTenantHeadersStripped:
 AM3PEPF0000A799.eurprd04.prod.outlook.com
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id-Prvs:
	d2dca5a3-62c2-4ffd-f314-08dc853ff526
X-Microsoft-Antispam:
	BCL:0;ARA:13230031|376005|82310400017|1800799015|35042699013|36860700004;
X-Microsoft-Antispam-Message-Info:
	=?utf-8?B?SkxzSTFmTWNaMHRBMnFHd1prcWpQamVlTVVqK29xWEptUzhIZmVZM3g2K2lr?=
 =?utf-8?B?anIwKzJabVJHOE1mK05UTHlwZ016bGFMdGJJQXQzYy9mOWNJUitUS2Q1ZXZO?=
 =?utf-8?B?SncvMFdFQ21UTCtUZVp1aTBRN1dhQitEeUgzUkZhaXpHUDVKT1pGbVNTS1B3?=
 =?utf-8?B?UlM1aEE5cmJJOG12NWUxVVpPcGdCWUxtaU5lcmhhWGVvdlNGUnpGQlNCclRj?=
 =?utf-8?B?YXQ3aTRhYUJYRDhOM1pQcTlST21RLy9QQjN2NlhPNXpiQ3lVRHljdldJUEVT?=
 =?utf-8?B?bFVUT1c2QnVybExBRTBIKzNiWkxoUk90SFFDd0lsQm9UMEFuUEUwVU9tT1Ur?=
 =?utf-8?B?YTJHcytMTVVIWjJmTlZaREY2dlBid3psN0NwRWNEQ2thdm85Rkp0U0xaWWY0?=
 =?utf-8?B?eXJZTXd4M3pGRXFPTTdpb2xVS3NMdTM5S2hObS9NUkJ3OUd3R25VQUR3bHlw?=
 =?utf-8?B?WVdybVBPQ3R3UHJ3ekJFekZUa3VZVnd2eWp3ZnRCR2dpb2R6K0lBYWYydFlY?=
 =?utf-8?B?OW1FOHdjTXIxbGVBazMwcnVrS1h0YVBkdTA2MTNmc1Bnd29iTjIvL0g3ODh3?=
 =?utf-8?B?RXY0UnpQeVhqSElhbWJNV0ZtUkF1bEplQzBJTEU2eDRCODhKbG96OFZPdUUx?=
 =?utf-8?B?NnlHR2Z5eG1wT3pZVW1kbUVpaDlCTVhLNWc3Ukk2QzVtUEpxWThLTzl2V0RV?=
 =?utf-8?B?Z3YxWjhuWnc0SUx4VUlBeEF4cCtkMW9pU2tvRGV4YlhqK0l2eGdEQTI1aDha?=
 =?utf-8?B?RzlkSjZINFZwVk90b1ZtN0g2RlFacUI3ckMwd2R5YjRWMDJrVG44elVxTlRn?=
 =?utf-8?B?NFlDSllkeGtpS2ZGd2ZDLzMyOUt1T1BiRjVZYVNqVm5FaDVvN0FOZFVNOUhr?=
 =?utf-8?B?VlRldkR2SWtlUjlNR01XVmgyUHJxRktyV3JnTkVlZ1hvcmF0M0Q5ZktUVGZP?=
 =?utf-8?B?N2dxVklJanVWYUUzUmk0aTZoWWp1TDZQUjlYVFVwTDdmTGRxYTVFaysxNCs4?=
 =?utf-8?B?MS8zbEhuZ3ZOd1hyNXRLbWVLOWNMUHBoUDFET2lYWDdnRWRtUHorU05zbHAv?=
 =?utf-8?B?MGVYemtxU2NycEtuQkdTSFlneG1TK0FLaGFoNmFOdkdGQW8wc0RkTFhIcit2?=
 =?utf-8?B?L2NkQUoweFh6L3dvSGNIQTNVSEtLb3hRU2U4TzR6Zm5uZDVSNzNYLzNaTVU0?=
 =?utf-8?B?OHVzeVc5dW9YKzk4L29wOERpeE9sM1F2Y2F6VjhxSWxyK2Eyc29BY2Y4V2d3?=
 =?utf-8?B?WmVDakErWm41QURhVHVySmo0M2ZIMGVvMHpESGY2T29SWklOUDhBWngvN0hT?=
 =?utf-8?B?VlczaWVTaEZiYi9KSFd6UkVuRjZLN2E3M1c4RlQxS1FRUVgrbktXcHlvdTY3?=
 =?utf-8?B?cEtBZzNOcGZYUVQxNHVZYzg3b1l3ZzdBa2M3eGE2d0N2TFpubDhDaG84WHNn?=
 =?utf-8?B?YkFFYXRsY0hOdUlBYkYwQ0lsMG51Zm8zamQzNE9sRm9LL0c4MjhXcXVUblU4?=
 =?utf-8?B?eFgydUdjaW9XQ0lzNVZkOGpXU2FKeGczZmJvZk1Tb1N6dkRyblBQU2ZMTUh0?=
 =?utf-8?B?dUJYK2FmdVVEcEEyM0V3TXo0ZWovNVpsREhuV0dTajdIL1VDZWJzbkFpa1Rs?=
 =?utf-8?B?dVBzM1Foc2xkQzNWQ1BMaE40TTlYdUlnWG1KMGtraTR2YmJFS2J2dW1PS0Uy?=
 =?utf-8?B?MVRMOU9ac1NTVERJMW4ySWVPbXdKNHZYOGIvejdQekNWTkp4cFErb2tRcnVF?=
 =?utf-8?B?S3oyazRNc21kUS9HOWQ3dHNFVEFrTXMzNDhmdmZHWkcwSk1tUG9oa0FYSVAw?=
 =?utf-8?B?UXVGbmdJZUs1M2xBYk5hY25Ic2xaZ3p6K1l1REFoM1lDZXZ0VFp0aUZCbW5W?=
 =?utf-8?B?M1FhSXlqMVBEQzdRMXY1R0JnR0tCYWsvYmYvV2J1QldXWXc9PQ==?=
X-Forefront-Antispam-Report:
	CIP:63.35.35.123;CTRY:IE;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:64aa7808-outbound-1.mta.getcheckrecipient.com;PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com;CAT:NONE;SFS:(13230031)(376005)(82310400017)(1800799015)(35042699013)(36860700004);DIR:OUT;SFP:1101;
X-OriginatorOrg: arm.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jun 2024 09:15:06.8224
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: e0273df1-be40-43e7-72fa-08dc853ffde7
X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d;Ip=[63.35.35.123];Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com]
X-MS-Exchange-CrossTenant-AuthSource:
	AM3PEPF0000A799.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR08MB10923

On 6/5/24 08:22, Peter Zijlstra wrote:
> On Tue, Jun 04, 2024 at 09:12:20PM +0200, Peter Zijlstra wrote:
> 
>> But with the above, you skip inc for sched_delayed, but dequeue_task()
>> will have done the dec, so isn't it then still unbalanced?
>>
>> Oh well, I'll go stare at this in tomorrow.

It did not appear to keep things unbalanced towards uclamp_rq_dec, but
might've been luck of not hitting the right code path.

In any case ...

> 
> OK, just before I went to play in my giant hamster wheel it hit me.
> 
> When dequeue_task() 'fails' and sets sched_delayed, we'll have done
> uclamp_rq_dec().
> 
> Then, since the delayed task is still on the rq -- per the failure -- it
> can be migrated, this will again do dequeue_task() which will *agian* do
> a uclamp_rq_dec().
> 
> So now we have a double dequeue -- *FAIL*.
> 
> Worse, the migration will then do an enqueue_task() on the new rq
> causing uclamp_rq_inc(). If you then get a ttwu() / ENQUEUE_DELAYED, you
> can tickle yet another uclamp_rq_inc() for another fail.
> 
> Something like the below avoids uclamp_rq_{inc,dec}() when
> ->sched_delayed, and moves it post class->enqueue_task() such that for
> the ENQUEUE_DELAYED case, we *will* do the inc after ->sched_delayed
> gets cleared.
> 
> Hmm?
> 
> ---
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 9ac1054c2a4bb..965e6464e68e9 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -1676,6 +1676,9 @@ static inline void uclamp_rq_inc(struct rq *rq, struct task_struct *p)
>  	if (unlikely(!p->sched_class->uclamp_enabled))
>  		return;
>  
> +	if (p->se.sched_delayed)
> +		return;
> +
>  	for_each_clamp_id(clamp_id)
>  		uclamp_rq_inc_id(rq, p, clamp_id);
>  
> @@ -1700,6 +1703,9 @@ static inline void uclamp_rq_dec(struct rq *rq, struct task_struct *p)
>  	if (unlikely(!p->sched_class->uclamp_enabled))
>  		return;
>  
> +	if (p->se.sched_delayed)
> +		return;
> +
>  	for_each_clamp_id(clamp_id)
>  		uclamp_rq_dec_id(rq, p, clamp_id);
>  }
> @@ -1979,9 +1985,10 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
>  		psi_enqueue(p, (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED));
>  	}
>  
> -	uclamp_rq_inc(rq, p);
>  	p->sched_class->enqueue_task(rq, p, flags);
>  
> +	uclamp_rq_inc(rq, p);
> +
>  	if (sched_core_enabled(rq))
>  		sched_core_enqueue(rq, p);
>  }
> @@ -2003,6 +2010,7 @@ bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)
>  	}
>  
>  	uclamp_rq_dec(rq, p);
> +
>  	return p->sched_class->dequeue_task(rq, p, flags);
>  }
>  

... thanks for the patch! The above seems to do it for me. I can see
more reasonable energy use with the eevdf-complete series. Still a
bit higher. Might be noise, we'll see.

I'll go stare at it and run some more tests on our end with this fix
applied on top and will report back.