Received-SPF: pass (google.com: domain of linux-kernel+bounces-4757-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <1c13b0cc-098c-4f75-9f37-a53a66965409@oracle.com>
Date: Tue, 19 Dec 2023 17:41:51 +1100
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC PATCH] sched: fair: reset task_group.load_avg when there are
 no running tasks.
From: Imran Khan <imran.f.khan@oracle.com>
To: Vincent Guittot <vincent.guittot@linaro.org>
Cc: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com,
        dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com,
        mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com,
        linux-kernel@vger.kernel.org
References: <20231215052652.917741-1-imran.f.khan@oracle.com>
 <CAKfTPtD85OSem=7RMquLWokVp7gffvDaY3mtwevkxp1mSSVVqQ@mail.gmail.com>
 <d8507f86-2458-4b01-a774-5102473e657e@oracle.com>
Content-Language: en-US
In-Reply-To: <d8507f86-2458-4b01-a774-5102473e657e@oracle.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?R2J1VEFYcTVxVkRFSlA1dVh2YWU3N1dqK1hCUG5uUVVBTExGVVdIckducURv?=
 =?utf-8?B?RVEyckIyYVhJV3dNbFVlZFBJMFd0VFZqQ3ZxeTA4a1dQdmxpMXNzZW03U3dP?=
 =?utf-8?B?d01Pd1VrZ0F2cklQWkpPbzRSRXNWNS9jeUVRQjljbTVnWTlzbDY0T041S2tY?=
 =?utf-8?B?U2ZncXBCTUg3QmhrUlRPOE9nMWQrUWhTaUIyMEw4Y2xub3lxdDA2ZlVBYnRR?=
 =?utf-8?B?UDJ6d2V6dUFwb2FKZW5pZU5QQXNjVDd2TTJLLzlWdnFDQUFoQml4QkIrbUlI?=
 =?utf-8?B?MnhBdUFrQkNFTU9WUTM0UitYaEh6cjJCL1NHNWw1OVBLV05ybVpHcGpwOS9D?=
 =?utf-8?B?M1FXRVJISlZSSkFqbDBUb2ZsNklZeG1wMVJJWVdzampmampLSWs5aHNNQkw2?=
 =?utf-8?B?Um05OXVEcnJtODM4MktLc1AweUIvMEJvQjVmK3VScHg2dGUzMEVSM1BIcEJR?=
 =?utf-8?B?VGtNdDQvMVlsY1Vnb3FsQTZUS3BtUkZUQ2xpeTY1UXRKMWo2UFZsSm5ITCtn?=
 =?utf-8?B?TVNqbmNLNW5kNk84SEREVmpTTHBxb3RkZ2lrdXl2MDZwRWxYNGI3d3Rrd2Y3?=
 =?utf-8?B?NkRXaGd0KzF1eUFVMU5JQ3k3NG9OQ28xYzV0VnptcldaYzhjclllUy9yTFp6?=
 =?utf-8?B?bTZTMmphYWZkN0FpbDIxZmxLNnN2V3YrRkF5ZDE1RDVMd2Q4Z2FwUlAzVEla?=
 =?utf-8?B?YXZKR1NOKyttN0owVFppSUJQMzNvZy9PWi9XTWkvOVVITCtLUjhocWp5UFFJ?=
 =?utf-8?B?TGIvV2luTTloQVY4NERYNzlTMHZSYUExeXJ2MzNnZ1A0MHJvRzROanpKSHUy?=
 =?utf-8?B?a3d3Y1I4QWhXb2o4QzJyWmlIa29XelVlRTJJZVlVanVJNVRoeWJlOTVTLzg3?=
 =?utf-8?B?T1JFM2RmbWNYT3M1WVRLSEdYZTRrbGZXVVYra3pwajRYQUVHanJrNTFEaHNn?=
 =?utf-8?B?NUJZVmtaNGluT1FBRVYzNTBVU1FKbSsvcEVzNWdVcENyWTA5UFdwMmlaZlZH?=
 =?utf-8?B?UjdaOXZVaC9TUklhc2xGL2VmZ1drMko2QU91K3FCQWRpT1dIUkZ3RzFVMjlw?=
 =?utf-8?B?T0MyOW5mTlI2Lzd4NzROb0F6aFo2OU1vUDZ4YXR3STB4azE5N2Y2TE1yR0ZH?=
 =?utf-8?B?Ri85cmU0TkxhWUYrVitMS0dselQwdjRHdXg2WWM3YVlqb3Nhc29kUXBFcWh4?=
 =?utf-8?B?ZDlUOWo3RjlXdXNUUmx1TFhLTS83MWlHVVY4aXhqNi9aTEFmbEJCRW1qMCtB?=
 =?utf-8?B?VGthNmJIdWdLT0lEUHU5TkEvUUlNWUl2SWdyZVZiOTdWc0FuL3B2aGx1NlB2?=
 =?utf-8?B?dlQzQkJUV0JoczNqSWlVSTlwanhBNzN2WUFNd3hwNTBQY2ErTmpNUUQxZVdN?=
 =?utf-8?B?N2F2WEJRRFB1aFdEc3R3dTErTndkTkRkYnZtcExTZ2ExYUtKU081aDROcUlY?=
 =?utf-8?B?b3lFeEs0WkVGSlZPZXhwek1Ud1k2aWZwd0RMRVdjVlFxM09kQXFIS1k1ZGNy?=
 =?utf-8?B?aVZwVUZyeWMzQmJiVy94RG9HM0hGd3Q2c3Y3cTgvYnhuMEVRRzhzODV2cVlQ?=
 =?utf-8?B?SmdxeHFiR0FkajRsWFpDaTA2WmkzdFpIRDNNdSt0Unh4bm9HdmJXVnJvWkFL?=
 =?utf-8?B?TFNSa09JVmxNQkJtMWpXRUh0MnErVEZ3TXcrNGpQb2Y3TTAyYTVZdnUxTG9i?=
 =?utf-8?B?cjl5UXFxREpGNjhhRDFrbDIyTVcvZ0FxU2N1dUlWN2NvL1A5QXIyeGx6T01T?=
 =?utf-8?B?ckwwa0E0emZFQzQ1czJSdFlFK3gvWlcvTEJLbGhXbjAyVmpiZkRUY0I3dXlm?=
 =?utf-8?B?ejRWbDQwR3pPS3Y5aUdwLzBtWVA5bEorYTVyVzRsMVpBR3RybUVMd1hnZk9o?=
 =?utf-8?B?SHVIWEk2MVc0eTM2cWk3SVFOd3BhdWRUTTIwaXc2R1hjcnZQWWpjTzJEaW9E?=
 =?utf-8?B?MHlmdDBIbmNjNkhXVWplc1Fwcm9rb3B1b1RjYWlTZXdPSTVFcjJLYlhUdE5R?=
 =?utf-8?B?NVAzdVpHYXUyMXFRNWlZVkhpWDNvQjBiRnEzMUJ4MWxZa1JLeHBsL1EvcVJW?=
 =?utf-8?B?RmZKR2ZNK284WDR0bzViMGg5ZWV3UGthU0ZBc0dWUnR0Mzl4emovZ29VaXUy?=
 =?utf-8?Q?ibhaz1s/QgnYdP80Mo8kS7jF5?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	vGipZywWJxlUEZPEi7X/Ym3Nyyr37DiDdG2UtAXMvCISzvXDNFYpajKqB2e3+4fN90XOZxMA+ND+qadzZ7lpOUS28XD5vXDtxh+Vlp8+xh58LZmwIrTOYiRmYICoZ4GpT5i5DwsPzOkByRvelEH9dUhkfdaqzQRwtsJIw1dsawL0ugbLmLiU3nY67+BgpGf7DoJScjwcqEATHhfTW6Ox8eMAnTj3lbrizswvybonaA/sS6v9wXDkMEB/OycrAlVVbqSmzVhv0IGu51VyIXN3DWEHQA/pHaDSrDIy8iSiJYu7BLQyXJvuIzEaJopGTCnGYsN7NOqNluNobaiOtqSYK3EqRzUpQJ8V+TC9+BBE8eWTQk2sQiXOAS12hlx1DWCCHj6OBAkgS9nYFfLQN8hih2z3e1K51c5WGm5cTw1AWyvkbtJ3SijoOPECRDJ24OyYNxk5DYkH9MWiH4hAOa/hH2syHxa6QTDrJxTxq8Aq+V47yV5Gusm6cf521aS0G7UHkLM78WkUiXtPHPudAImz662uSrAbiSMnF1I4WU9qzKtcYsuvO931GT1rnYgMoFFeHNqv+/06i5Mbsv32tGFicZI8BGasbPvQDUo0/nJ1mmE=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 726feaad-00c1-4d81-7610-08dc005d9a64
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4479.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 06:42:01.1140
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ZHW14JnMfKvnutpBhqwx/IjI6ppbqBrQUSSA8/g/eB8NmusU+vq/NJqEgYlhGT680lkY0mGZ66CPtyxzLtP7xQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6751
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-19_02,2023-12-14_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 spamscore=0
 mlxlogscore=999 adultscore=0 suspectscore=0 mlxscore=0 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2312190047
X-Proofpoint-ORIG-GUID: 76SyY8QB4KJf2YZnzEnhEYEwSudmhH6E
X-Proofpoint-GUID: 76SyY8QB4KJf2YZnzEnhEYEwSudmhH6E

Hello Vincent,


On 15/12/2023 8:59 pm, Imran Khan wrote:
> Hello Vincent,
> Thanks a lot for having a look and getting back.
> 
> On 15/12/2023 7:11 pm, Vincent Guittot wrote:
>> On Fri, 15 Dec 2023 at 06:27, Imran Khan <imran.f.khan@oracle.com> wrote:
>>>
>>> It has been found that sometimes a task_group has some residual
>>> load_avg even though the load average at each of its owned queues
>>> i.e task_group.cfs_rq[cpu].avg.load_avg and task_group.cfs_rq[cpu].
>>> tg_load_avg_contrib have become 0 for a long time.
>>> Under this scenario if another task starts running in this task_group,
>>> it does not get proper time share on CPU since pre-existing
>>> load average of task group inversely impacts the new task's CPU share
>>> on each CPU.
>>>
>>> This change looks for the condition when a task_group has no running
>>> tasks and sets the task_group's load average to 0 in such cases, so
>>> that tasks that run in future under this task_group get the CPU time
>>> in accordance with the current load.
>>>
>>> Signed-off-by: Imran Khan <imran.f.khan@oracle.com>
>>> ---
>>>
>>
>> [...]
>>
>>>
>>> 4. Now move systemd-udevd to one of these test groups, say test_group_1, and
>>> perform scale up to 124 CPUs followed by scale down back to 4 CPUs from the
>>> host side.
>>
>> Could it be the root cause of your problem ?
>>
>> The cfs_rq->tg_load_avg_contrib of the 120 CPUs that have been plugged
>> then unplugged,  have not been correctly removed from tg->load_avg. If
>> the cfs_rq->tg_load_avg_contrib of the 4 remaining CPUs is 0 then
>> tg->load_avg should be 0 too.
>>
> Agree and this was my understanding as well. The issue only happens
> with large number of CPUs. For example if I go from 4 to 8 and back to
> 4 , the issue does not happen and even if it happens the residual load
> avg is very little.
> 
>> Could you track that the cfs_rq->tg_load_avg_contrib is correctly
>> removed from tg->load_avg when you unplug the CPUs ? I can easily
>> imagine that the rate limit can skip some update of tg- >load_avg
>> while offlining the cpu
>>
> 
> I will try to trace it but just so you know this issue is happening on other
> kernel versions (which don't have rate limit feature) as well. I started
> with v4.14.x but have tested and found it on v5.4.x and v5.15.x as well.
> 
I collected some debug trace to understand the missing load avg
context better. From the traces it looks like during scale down,
the task_group.cfs_rq[cpu].avg.load_avg is not getting updated
properly for CPU(s) being hotplugged out.

For example if we look at following snippet (I have kept
only the relevant portion of trace in the mail), we can see that,
in the last invocation of update_tg_load_avg for task_group.cfs_rq[11]
both the load avg and contribution of this cfs_rq were 1024.
So delta was zero and this contribution eventually remains undeducted.
In this case scale down was done from 16 to 8 CPUs, so CPU 11 has been
offlined.


cpuhp/15-131605  [015] d...  6112.350658: update_tg_load_avg.constprop.124:
   cfs_of_cpu=5 cfs_rq->avg.load_avg = 0, cfs_rq->tg_load_avg_contrib = 0
   tg->load_avg = 0 delta = 0 ###
 systemd-udevd-894 [005] d...  6112.351096: update_tg_load_avg.constprop.124:
   cfs_of_cpu=11 cfs_rq->avg.load_avg = 1024, cfs_rq->tg_load_avg_contrib = 0
   tg->load_avg = 0 delta = 1024 ###
 systemd-udevd-894 [005] d...  6112.351165: update_tg_load_avg.constprop.124:
   cfs_of_cpu=5 cfs_rq->avg.load_avg = 10, cfs_rq->tg_load_avg_contrib = 0
   tg->load_avg = 1024 delta = 10 ###

.........................
.........................
 cat-128667  [006] d...  6112.504633: update_tg_load_avg.constprop.124:
   cfs_of_cpu=11 cfs_rq->avg.load_avg = 0, cfs_rq->tg_load_avg_contrib = 0
   tg->load_avg = 3085 delta = 0 ###
.........................
 sh-142414  [006] d...  6112.505392: update_tg_load_avg.constprop.124:
   cfs_of_cpu=11 cfs_rq->avg.load_avg = 1024, cfs_rq->tg_load_avg_contrib = 0
   tg->load_avg = 4041 delta = 1024 ###
.........................
 cat-128667  [006] d...  6112.504633: update_tg_load_avg.constprop.124:
   cfs_of_cpu=11 cfs_rq->avg.load_avg = 0, cfs_rq->tg_load_avg_contrib = 0
   tg->load_avg = 3085 delta = 0 ###
..........................
 sh-142414  [006] d...  6112.505392: update_tg_load_avg.constprop.124:
   cfs_of_cpu=11 cfs_rq->avg.load_avg = 1024, cfs_rq->tg_load_avg_contrib = 0
   tg->load_avg = 4041 delta = 1024 ###
..........................
 systemd-run-142416  [011] d.h.  6112.506547: update_tg_load_avg.constprop.124:
   cfs_of_cpu=11 cfs_rq->avg.load_avg = 1024, cfs_rq->tg_load_avg_contrib = 1024
   tg->load_avg = 3010 delta = 0 ###
..........................
 systemd-run-142416  [011] d.h.  6112.507546: update_tg_load_avg.constprop.124:
   cfs_of_cpu=11 cfs_rq->avg.load_avg = 1024, cfs_rq->tg_load_avg_contrib = 1024
   tg->load_avg = 3010 delta = 0 ### <-- last invocation for cfs_rq[11]

..........................
..........................
<idle>-0  [001] d.s.  6113.868542: update_tg_load_avg.constprop.124:
   cfs_of_cpu=2 cfs_rq->avg.load_avg = 0, cfs_rq->tg_load_avg_contrib = 0
   tg->load_avg = 1027 delta = 0 ###
<idle>-0  [001] d.s.  6113.869542: update_tg_load_avg.constprop.124:
   cfs_of_cpu=2 cfs_rq->avg.load_avg = 0, cfs_rq->tg_load_avg_contrib = 0
   tg->load_avg = 1027 delta = 0 ###
 <idle>-0 [001] d.s.  6113.870541: update_tg_load_avg.constprop.124:
   cfs_of_cpu=2 cfs_rq->avg.load_avg = 0, cfs_rq->tg_load_avg_contrib = 0
   tg->load_avg = 1027 delta = 0 ###


If I understand correctly, when CPU 11 is offlined the task(s) on its cfs_rq
will be migrated and its cfs_rq.avg.load_avg will be updated accordingly. This
drop is cfs_rq.avg.load_avg will be detected by update_tg_load_avg and hence
the contribution of this cfs_rq will get deducted from tg->load_avg.
It looks like during hotplug load of one or more tasks, being migrated are
not getting accounted in the source cfs_rq and this is ending up as residual
load_avg at task_group (if these tasks are members of a task_group).

Moreover this looks racy and dependent on number of CPUs or some delay.
For example for scale down from 124 to 4 CPUs I always hit this issue but
for scale down from 16 to 4 CPUs I hit this issue 8-9 out of 10 times.
Also for the cases when residual load_avg at task group is low (like < 10),
I can see that both of my test cgroups get similar CPU times which further
proves that the unaccounted load avg ending up in a task_group is eventually
leading to uneven CPU allotment between task groups.


I am debugging it further but in the mean time if you have some suggestions or
need traces from some specific portion of sched code, please let me know.

Thanks,
Imran

> Thanks,
> Imran
>