Received-SPF: pass (google.com: domain of linux-kernel+bounces-200809-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Received-SPF: Pass (protection.outlook.com: domain of arm.com designates
 63.35.35.123 as permitted sender) receiver=protection.outlook.com;
 client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com;
 pr=C
Authentication-Results-Original: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=arm.com;
Message-ID: <24e09046-74ee-4ebb-ac1a-bdc84568e825@arm.com>
Date: Tue, 4 Jun 2024 15:23:41 +0100
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC][PATCH 08/10] sched/fair: Implement delayed dequeue
Content-Language: en-US
To: Peter Zijlstra <peterz@infradead.org>
Cc: mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org,
 dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com,
 mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com,
 linux-kernel@vger.kernel.org, kprateek.nayak@amd.com,
 wuyun.abel@bytedance.com, tglx@linutronix.de, efault@gmx.de, nd
 <nd@arm.com>, John Stultz <jstultz@google.com>, Hongyan.Xia2@arm.com
References: <20240405110010.631664251@infradead.org>
 <3888d7c8-660e-479c-8c10-8295204e5f36@arm.com>
 <1461277e-af68-41e7-947c-9178b55810b1@arm.com>
 <20240425104220.GE21980@noisy.programming.kicks-ass.net>
 <20240425114949.GH12673@noisy.programming.kicks-ass.net>
 <20240426093241.GI12673@noisy.programming.kicks-ass.net>
 <c6152855-ef92-4c24-a3f5-64d4256b6789@arm.com>
 <2fba04b0-e55e-41f4-8b7a-723734fe1ad2@arm.com>
 <20240529225036.GN40213@noisy.programming.kicks-ass.net>
 <7eac0774-0f9d-487c-97b6-ab0e85f0ae3a@arm.com>
 <20240604101107.GO26599@noisy.programming.kicks-ass.net>
From: Luis Machado <luis.machado@arm.com>
In-Reply-To: <20240604101107.GO26599@noisy.programming.kicks-ass.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
NoDisclaimer: true
X-Microsoft-Antispam-Message-Info-Original:
 =?utf-8?B?dHJYRWNtQ1I5eWpaaDFhUE5xLy9oK0dsWW1FY0kvZEhUckxNRUdaQmhtTSs2?=
 =?utf-8?B?TnV1RDViNU80aU0vVlBRaEoybGpxZEVJTUZSbklqRktRSkJqVnROME9Gemlx?=
 =?utf-8?B?dThKS214RG1KcHVzMkZrYllBMitESHloR1Q2YnM4TjdYckFqWXRuSllRTUdu?=
 =?utf-8?B?emJvS1o5eGg0OG5adGtueGxIS3VER0pIeTZkT2pLMS9sNVMxemx2UHFyTHRD?=
 =?utf-8?B?NGFKSGRUSWRMM2pDYmY1cnBERktvNXNOS2VSN2ZOWnlCMjgwR0ZyaGhRT0ww?=
 =?utf-8?B?L0JrYjJscndRZFo1ZWVuMHErcDZkbFlSYWZyUGt4SkxjK29TbmJ2Y0tuYUdS?=
 =?utf-8?B?NlZkVmxnNzJYc0krU0RqNFNkN1Zxd2ZZR3NqcDRUOTR1MHp1Y25sVkM2TDZU?=
 =?utf-8?B?aW9EQUpXbENXZTVZWjA4Z09KYUxKWHFITHZKRTh4ZktRcTJkT25pVzBCQkVu?=
 =?utf-8?B?aVhaZkVrNmhEak9QVWZ1cjhLaVhTWURkMDlrZnlManhtQ2tORlYzM0pvZVd5?=
 =?utf-8?B?MVE5dmgzbDEwZ3lpRUNpWHNQN1BseXdoUkRLRnltVEVQWVQ1UU04UWF3MzEv?=
 =?utf-8?B?MEVNSEVkc25zQTJpWXFoUHpYakFRTUNmdmJLYmhrT05yb2sycDhxaXBBV0FC?=
 =?utf-8?B?ZE5BNjBxWjY3N0J6ZlBkeHBsTFMraWsvT1NhOUpMMFMxL1lDVnZ3YlFZN0RC?=
 =?utf-8?B?ZkdPYzRzUFIwcjI2eSt2em8wMVpjcnVTWTQrQWtqb3dJSkJMak1aUmZhWUpQ?=
 =?utf-8?B?enFkUkFnZjVtekE0MUNoK0hEMVVBNG1RM1RQRjJoRnZFSUNDdVNpNVlua0dT?=
 =?utf-8?B?bjhDOWd5MHlOOUZZazE4dURMV2k1ZC9yZVNFNTJObWI1RlpnS2tsci8za2dT?=
 =?utf-8?B?aWlKQ2g5aE9ZcEgrelpJS2NCZ2lxNHoxemc0WWM0OU41eithT3NVZ3o3M3dy?=
 =?utf-8?B?WHUrbmx6c0RVUm9XMnhyMFlUSzQ4SjVKYUMvd3NTR09pUUlKVWkwdExPY0I3?=
 =?utf-8?B?RUNxQzJPNHpXS0FtcFZMNjlXVGxsakhORFBScklCOUU5ZmIwa2JpV2RITUFC?=
 =?utf-8?B?NHJnOVFLNC9BSE9mdmlTN2hzcHFZVjVMTktFLzJRWlVJWjhhTVIwd29UeFJR?=
 =?utf-8?B?d0JTNlM2cGRVWGxRT25FbGtPQlEvRDRWZEtvWCtORVVVSGptaUxxSjhINkVi?=
 =?utf-8?B?NUIyZUUxTXhsL0pmLytwUHpDZUpXL0JHZTUxeVlYTTdVNi9HdXovOFVqYVVU?=
 =?utf-8?B?d0x2Q3VzTWlHWUY0Umhia0YzMXgvWnMwMU9DTnVaR3FCNk9SR2MzVlBoRENK?=
 =?utf-8?B?NThRMzBrSXYrc0thMmNTZ2thRGJqVXBmSVc5WHZieG5aWmcvelo2RkhBN0pl?=
 =?utf-8?B?N1c0ekhaMUwyKzMrMGdrUzdRZkRPVC9VQjRybENjY0phcWMrU0hYRk1JdDJR?=
 =?utf-8?B?cW5ZekZoQWhpQ0RDOXhzMUtIbFhPTytuR0FRckpwL1ZWekRkTkQ2SFZhNyt1?=
 =?utf-8?B?S3IyY1cvM3M3V3kyM3VHT1NjVjEvbnV6RDBVdXYreFF3RVFVaEYrRkpVdUhy?=
 =?utf-8?B?aGdzTWgrMHVFWWtYQVlxaTFxTmtjdkQ4bUpGdlpxdjg5S25ybVJZMGNBV1NG?=
 =?utf-8?B?K09VTE1HVmlSV0NFNFdaMDJFMWhnVFVXWEJqV3Y1eE9OS3V0Q0diY2tOOXRm?=
 =?utf-8?B?L1ZFVGI5US9jWDA5UmdEN09lMm9ldytPN1RXdm9vYkRUeVRBR0d0VEdBPT0=?=
X-Forefront-Antispam-Report-Untrusted:
 CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR08MB3919.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(366007)(376005)(7416005);DIR:OUT;SFP:1101;
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR08MB6303
Original-Authentication-Results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=arm.com;
X-EOPAttributedMessage: 0
X-MS-Exchange-Transport-CrossTenantHeadersStripped:
 AMS1EPF00000046.eurprd04.prod.outlook.com
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id-Prvs:
	6eb604d9-10a6-4b12-bf84-08dc84a1f18d
X-Microsoft-Antispam:
	BCL:0;ARA:13230031|1800799015|82310400017|35042699013|36860700004|376005;
X-Microsoft-Antispam-Message-Info:
	=?utf-8?B?K25WU1JUcGdTVEdjRTNxb290ZUNKQlozQnc0cHE5RjFMNFE5TnphZ0wyNzht?=
 =?utf-8?B?b0NTRjZ0NmIwOXY3Q2g1T1h2dEVPQWxCakVJSzJ2eUNaSTY5Um5jNlZWRjZZ?=
 =?utf-8?B?Q04rajgzd2hGMHY3c2RoNHBVeFlKNHF3Zm5KN2VRWEJwNnhaSCtRUzl2SE5i?=
 =?utf-8?B?cTE5bHFCZi9GNWl1elloc1ozS2Zra3FoT1ptVEhUVDZ1MTFJK1N1RjZEaFEz?=
 =?utf-8?B?eG80U0R0enhlZlNhemdTZTlVTkIwQk5xcmV4NDhwZWdZQ05vZGRhYnVUaVBp?=
 =?utf-8?B?d1hoZlV4VFBVeWpIYkJtc0ZLZEFVZTk2dnpFaUdYb3ZJUnVsUWlFWjVrMW5E?=
 =?utf-8?B?NjF0R1hlNFBzZ0ZxaWdaYmJOU2RqemtMVjFxYXBnY0RZKzBKa3YvdWFIa1l6?=
 =?utf-8?B?VGdHTjBHYUVSMEQ4RXhmc1lGOTlFODlwTm84SHZwN0xjZW9mdmlpK1U0Q3Yz?=
 =?utf-8?B?bmFpck1rckRhaXpKby9tSVZvY3VSeWtpSzNLdGJaN1Y3bzg5cCtodU1XSit0?=
 =?utf-8?B?RDRaNEIzVjkxRDA0RzE2ZDE3clRTcnZhL2FLUDJ3LzlGR1F2bXJKRE84d0dE?=
 =?utf-8?B?empQb0RyY3pRZXk4b0MyUksxM3luV3ovK25oVGlTTmZxWTRRcGZDeGN2ZUFV?=
 =?utf-8?B?Mi91SWhjSUtCbEdtNm1aWVNZSHNDb1JuTnVZcEZzNEFrYXloYUU5U1B1anBH?=
 =?utf-8?B?UDZuTGQxSnQ4MjRtN3FnbHhVYUNkcHgrRlkvdGVIMkI0ZWh5UlNUY1FSQ1VY?=
 =?utf-8?B?blNZanphd2hTM21vcEYzak92eFN6OWFLUUR5b3dlK3E4S2pKd3h1elRmVXZD?=
 =?utf-8?B?akRMRmFIcHNxM2xJdTlIQ3F4eVRXK29kaTZiYmxvRmp4c3lhZ3VkZ2ozd2RK?=
 =?utf-8?B?eWM0ZG5aQ1daZnc5MTVadUhubGNyQU9IUWUwRlJ5c2FJM0haeEdZNFExMHFh?=
 =?utf-8?B?b0M5WDlHWEFaNVhxZzJZaHFrbVMybnBad3VIdmVVZERnWmszU250djQvdEJz?=
 =?utf-8?B?dFpjOHBtdU1DUlNObVM4SVVxZzRVa2gzNG9tVmlaakdNZzRzOGdtWWRWdnZr?=
 =?utf-8?B?ZWtEditWSzJmL0phbnVuN2cyeVZPSXJ0aGcwSEhxYVI4WmV0NHJDVjY1enBD?=
 =?utf-8?B?b2lRd1ozRmtDSklzdmhzU2swbUNERDltS2pKMWhHM1E4T0EySCtuR25ucVp0?=
 =?utf-8?B?R21QRFRRMEhSdlhJVWZHa3o4T2NVbGQ3N3dZdHQyeFlWVjZyZE9KR2p3UWZW?=
 =?utf-8?B?QlpxRVFUL051OVM2L1kzemUyendITkQyNjVQc29qRVFjWC81N3pwdmxodGVx?=
 =?utf-8?B?MXpXNU9CSHJjUVRoRlEwQXpBYmVuaERxVjhDZTc1M2d4eXUyVGhwekRxODFB?=
 =?utf-8?B?YTRNbTEydzJRQThobHNaVGR3dy93ME1sVUY1b1docFQxbEdCTVZCcnhrMVVJ?=
 =?utf-8?B?d0trdVZKQXF3UHJDU0FzdWZXTGxwcmF0VFVtZWVjcmVmeVRDdXNKOHplbXFq?=
 =?utf-8?B?OVhOd1J3RmJKbm1tVzVEMWNHZnVhajYvR2lwS1FselRMSlRhYUpoaFNpcDda?=
 =?utf-8?B?UUlXb2hra25wSHhlTlcrdk4yK1BrQTByWjhmR0Y1bkxUSE84OG53WHFSLzk2?=
 =?utf-8?B?aUNsQ2l3R1pyemlGclFkeWM3UVZnZ0xXbTJiNU55WjlWcWFrT2JhYlFXQ3NH?=
 =?utf-8?B?cldCQlMveW00SlFOd0hIQjl5TXpDS3ZrWVl4aFNVVlVIZ0R1eFVUck9LUEJ6?=
 =?utf-8?B?WTdWdmFaRGJYaDlNUFNlQUkxVFFaSHJtMUVqdnZ0WklXMkhkWHFad1A2cGZx?=
 =?utf-8?B?WndwMUEvS3pLRnV0a1oxVDlON0ZBYUcvaVEvV0luNWc2a1kzMTRFejVhR2lj?=
 =?utf-8?B?ZmUrVlBrV0xVZnVEb2JIUnNKRVdFV1FmaUZZUTlJVlNOUFE9PQ==?=
X-Forefront-Antispam-Report:
	CIP:63.35.35.123;CTRY:IE;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:64aa7808-outbound-1.mta.getcheckrecipient.com;PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com;CAT:NONE;SFS:(13230031)(1800799015)(82310400017)(35042699013)(36860700004)(376005);DIR:OUT;SFP:1101;
X-OriginatorOrg: arm.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jun 2024 14:24:01.1701
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: c6b972a5-10c9-4b9b-54c6-08dc84a1fad0
X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d;Ip=[63.35.35.123];Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com]
X-MS-Exchange-CrossTenant-AuthSource:
	AMS1EPF00000046.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS4PR08MB8144

On 6/4/24 11:11, Peter Zijlstra wrote:
> On Mon, Jun 03, 2024 at 08:30:43PM +0100, Luis Machado wrote:
> 
>> Exchanging some information with Hongyan today, he was a bit suspicious of the uclamp
>> behavior with the eevdf complete series applied.
>>
>> Checking the uclamp code, I see we have some refcounting tied to enqueuing/dequeuing
>> of tasks, and the uclamp values are organized in buckets.
>>
>> Just for fun I added a few trace_printk's in uclamp_eff_value, uclamp_rq_inc_id and
>> uclamp_rq_dec_id.
>>
>> Booting up the system with delayed_dequeue disabled and running the benchmark, I 
>> see the uclamp bucket management pretty stable. Tasks get added to the uclamp
>> buckets but then get removed. At the end of the benchmark, the uclamp buckets
>> are (almost always) clean of tasks.
>>
>> Enabling delayed dequeue, I can see the uclamp buckets slowly filling up with
>> tasks. At the end of the benchmark, I see uclamp buckets with 30, 40 or 50
>> tasks still. If I do another run, I can see 80, 100 tasks still.
>>
>> I suspect refcounting might be going wrong somewhere due to delayed dequeue
>> tasks, but that's more of a guess right now. Hopefully that is useful
>> information. I'll resume investigation tomorrow.
> 
> Thank you both!!
> 
> Does the below help?
> 
> Note how dequeue_task() does uclamp_rq_dec() unconditionally, which is
> then not balanced in the case below.
> 
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -3664,6 +3664,7 @@ static int ttwu_runnable(struct task_str
>  			/* mustn't run a delayed task */
>  			SCHED_WARN_ON(task_on_cpu(rq, p));
>  			enqueue_task(rq, p, ENQUEUE_DELAYED);
> +			uclamp_rq_inc(rq, p);
>  		}
>  		if (!task_on_cpu(rq, p)) {
>  			/*

As Hongyan pointed out in a separate message, the above makes things
worse, as we end up with even more leftover tasks in the uclamp
buckets.

I'm trying a fix in kernel/sched/core.c:enqueue_task that only
calls uclamp_rq_inc if the task is not sched_delayed, so:

-       uclamp_rq_inc(rq, p);
+       if (!p->se.sched_delayed)
+         uclamp_rq_inc(rq, p);

I'm not entirely sure it is correct, but it seems to fix things,
but I'm still running some tests.

With the current code, given uclamp_rq_inc and uclamp_rq_dec get
called in enqueue_task and dequeue_task, the additional enqueue_task
call from ttwu_runnable for a delayed_dequeue task may do an additional
unconditional call to uclamp_rq_inc, no?