Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <0e42c46a-ccc4-e793-00b8-ae407e06846f@amd.com>
Date:   Wed, 23 Feb 2022 09:33:59 -0600
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.6.1
Subject: Re: [PATCH] kernel/sched: Update schedstats when migrating threads
Content-Language: en-US
To:     Phil Auld <pauld@redhat.com>
Cc:     Peter Zijlstra <peterz@infradead.org>, juri.lelli@redhat.com,
        vincent.guittot@linaro.org, mingo@redhat.com,
        dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com,
        mgorman@suse.de, bristot@redhat.com, linux-kernel@vger.kernel.org
References: <20220126152222.5429-1-carlos.bilbao@amd.com>
 <YhYKL4hxx4TNKHGD@hirez.programming.kicks-ass.net>
 <aac8f860-c01f-bda0-9f1b-029b234213c2@amd.com>
 <YhZSqd+d03oWUPP6@lorien.usersys.redhat.com>
From:   Carlos Bilbao <carlos.bilbao@amd.com>
In-Reply-To: <YhZSqd+d03oWUPP6@lorien.usersys.redhat.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MUsxTHJ3NFI1bEY4TEF6VitDeWZtaDI0a01xakpzNS9oQStDNDROUFJGZ1Zk?=
 =?utf-8?B?T3J6eEg0YXFvQ25iL2VyV08rM1U1ekh1Zk1DWUVKWlBBbDJtZGg1QTRvUHUv?=
 =?utf-8?B?S0NSaVNQelRrMHQ1ekFFQ1hsNUFhbkl1THZxWWYrc3lNdXUwTUZiRjBQZHFX?=
 =?utf-8?B?aFdJaDdIYXFBNEhzaU9VWWNyYkN5NGdKd3d1SnVUODhoSFh3SXJLSGd3Nkxa?=
 =?utf-8?B?aFU5NG5kMzVqQkVYSXh4T0dQSVJxaDl0RU9JMmhZRkZHQ2daOStQb3o5M2NC?=
 =?utf-8?B?SDB6ejkrakZrVHFIeW9tM2VBVHFhc2xaQmg1TWRtQ3lxN2o1TmhYVEZuN0pt?=
 =?utf-8?B?VzltbVY3MzVHOURjOUJlaENhM2VvcHZmZklVbFBPSU1yRUpxSzJKSGtlZk1k?=
 =?utf-8?B?OWVYTjhLWVZNT0UzS1ZJQTRWNCtJZEErNysranFITG52VllVOEZoWi9BVUZv?=
 =?utf-8?B?RWg0QitnYy9VdnR1c0NWQ01ZbXdaQU0xYmZCUGFUSkx6K1lvTS9CbFc1YmpL?=
 =?utf-8?B?eHZhR0ZGTDFvVVF3TUxBcE9LUjI3dUR5NUdQbzY3aTlWNlZKTDdaNXptTThC?=
 =?utf-8?B?K1JldUhHa3R5d20zYUxOMDlEalEwYkRmNUZ1cW5QaGJxVWFDWG1YRXk0ZGFj?=
 =?utf-8?B?NllrN28xb1JZNDVZTnpVZC9nelV1amJmcWtnajI0NWpocFA0amUwTTVUd3ho?=
 =?utf-8?B?cXN6bjRIWDJWME1vVjNBdGVNZzkraHhLNWlqVHZvT3ZybmRJTTJvSHArZ3Fr?=
 =?utf-8?B?YzZydlorTEpnc0V2dEZYRHJQZzI2VVNiZlVHS1ZjMjNBbnh5SEhHWEVkTm1D?=
 =?utf-8?B?cTZSbk1nSVpCdHZrME5aMXVpdUdZdndnck9iUUdETENrQTZBamQ2T1hJemIz?=
 =?utf-8?B?OVhlL1pVbXJoY05ybkhMNHJtd29heitGTnBLVjAzek9qVUdFYm5UdmVQTHk5?=
 =?utf-8?B?bXRjeXgrL3ZzeTdOOWdqSEpLbzVXRHdZL1o4Z29NUVh6Wk40MnFwUWloUWVm?=
 =?utf-8?B?bGxRQS9hUmdPNE1hWHB1VkFRZXJCRVJhaFEzUHJDVUxPd2diR2RyOE9UeDdN?=
 =?utf-8?B?UDZISCt4bnQvNFhGNjdQeVVXSVdRSG0wSWJ6Y1p4NkhhR3ZJUzdzSEFZUlhG?=
 =?utf-8?B?MnpDZzBPSHVZTlFwVXEwQkZ0c21VWEZUZWNVUzg1T0RIaGZpRERtYm51L1pO?=
 =?utf-8?B?d2ZGMVZmK1RlUkdRV0tTUTVuTlZNU1BocjBZL2pNdmZFREJMeWZVQVAvWjQ0?=
 =?utf-8?B?aGZ1b0NUc1dHZFEzYWRuajg0Z0ErR0h6SU1sYTlqeVU5K0dLbWxTQy9yazZt?=
 =?utf-8?B?SDkyUHpLei9QbmFxcUtPY3JRbGR1dVhJeEhPZmwvU01jV0E3RlJRbWFvM3o5?=
 =?utf-8?B?WXRhSkxjem9ia3ZTYWpiN1Rsa1pMUVlOQ3VlTUVZYTlnNVJNMG4yOGw1a1NB?=
 =?utf-8?B?K3I1UExKRnNyZU1EQm1KdXhiOU5wZlhKWUVid3M0dkNRcUZpeTZwc1czcUdz?=
 =?utf-8?B?bXJkQmRtK1BSZUQvUEJieUtXMjg0Qk5jOVVHZGhlaUFuUWs4cFYyZnQvZTZ1?=
 =?utf-8?B?TXNWdjV1L3dpZXJ2TjB1Q2pmMlF1Q05XUDlUbklhM3JMNkpDT3JDSHJYa3Ax?=
 =?utf-8?B?Z3pETkFSQnpZZi92b3VjWkhCczNHMmxLY3ZQcDJBVHRjSEVkbk05bU00c0t6?=
 =?utf-8?B?UkVja210dlhhak51akZLTGZtdGNUWUFsYkVwYVV5TTZJbk42NVozT2RMYWxL?=
 =?utf-8?B?L0xva3hvV3czWWNGVkNSejl0ME1hcGJ6T29MRXdkWUtaNlhtcU1LdnV6M2xk?=
 =?utf-8?B?MllPSnd0RlRmdGVTRlJZTEc0VnlJbkI5V0N1d0Nla3orWjE2NEpsY1FqR2Nk?=
 =?utf-8?B?Skd6d1Z0Yzhidnc4bzA1dzFZVENJQmVsTGZva1BuSVc5bFROeUkwR1g2UHRj?=
 =?utf-8?B?YXVzcEdpbVlpSE8xUkRyRktXNXlobnhmVGN1QVdWSjkvWTEyMS9hUTd6SzJu?=
 =?utf-8?B?U1JHZlBLK3VrZ3lWOWM4czgvVHhpeFBkRGZjTFdvOFdZajU5NzBHOHRKMkpC?=
 =?utf-8?B?b0REblpLU0xvenhEVWFLL1dDWUdpMUNWdms0WCtHTHBTMzZLQ2V0SXlacXJL?=
 =?utf-8?B?djJLL0Z6dnBUY1Z1VEdqcXVBRVl1QzZJWGZBNklvRDRQdHA4OEtiaXAxM05Z?=
 =?utf-8?Q?m6fiAemV6tRd5VywFOT2B8c=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9123018d-eef2-4d5a-d6a4-08d9f6e1eb4c
X-MS-Exchange-CrossTenant-AuthSource: BL1PR12MB5874.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2022 15:34:02.6797
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: CzDujFjLxph9GqqirBxbqZbrzW/f6bcKYBTdfZdUEAfNJO3jdmP4VQC1aVFwLHIF7NHFEvjd9MeqHrwuIF3YKA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR1201MB2522
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 2/23/2022 9:28 AM, Phil Auld wrote:
> On Wed, Feb 23, 2022 at 09:14:45AM -0600 Carlos Bilbao wrote:
>> On 2/23/2022 4:19 AM, Peter Zijlstra wrote:
>>> On Wed, Jan 26, 2022 at 09:22:23AM -0600, Carlos Bilbao wrote:
>>>> The kernel manages per-task scheduler statistics or schedstats. Such
>>>> counters should be reinitialized when the thread is migrated to a
>>>> different core rq, except for the values recording number of migrations.
>>>
>>> I'm confused, why should we reset schedstats on migrate? I'm thinking
>>> this breaks per-task, since tasks tend to bounce around quite a lot.
>>>
>>
>> Thanks for your comments, Peter. 
>>
>> Looking at the documentation of schedstats I see that most values are 
>> actually linked to the particular CPU: time spent on the cpu, timeslices 
>> run on this cpu, number of times _something_ was called when the cpu was 
>> idle, and so forth. Those values lose their meaning after migration and we 
>> should reinitialize their counters. However, reviewing sched_statistics I 
>> identify two fields that we should definitely keep increasing even after 
>> migration (nr_migrations_cold, nr_forced_migrations).
>>
> 
> The documentation is a little off. I think it should say "any cpu" instead
> of "this cpu".  If you reset these per task counters (time on cpu, number
> of timeslices etc) on every migration then they become meaningless (and
> useless).
> 
> 
> Cheers,
> Phil
> 

Well that clarifies it! Then, let me ask the opposite question... What
fields of schedstats should we clear when migrating? If there isn't any,
I will just increase the number of migrations.

>> So this patch will have to be upgraded if there's some other value(s) in
>> schedstats that we do not want to reinitialize either.
>>
>>>> Signed-off-by: Carlos Bilbao <carlos.bilbao@amd.com>
>>>> ---
>>>>  kernel/sched/core.c | 10 +++++++++-
>>>>  1 file changed, 9 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>>>> index fe53e510e711..d64c2a290176 100644
>>>> --- a/kernel/sched/core.c
>>>> +++ b/kernel/sched/core.c
>>>> @@ -8757,6 +8757,7 @@ bool sched_smp_initialized __read_mostly;
>>>>  int migrate_task_to(struct task_struct *p, int target_cpu)
>>>>  {
>>>>  	struct migration_arg arg = { p, target_cpu };
>>>> +	uint64_t forced_migrations, migrations_cold;
>>>>  	int curr_cpu = task_cpu(p);
>>>>  
>>>>  	if (curr_cpu == target_cpu)
>>>> @@ -8765,7 +8766,14 @@ int migrate_task_to(struct task_struct *p, int target_cpu)
>>>>  	if (!cpumask_test_cpu(target_cpu, p->cpus_ptr))
>>>>  		return -EINVAL;
>>>>  
>>>> -	/* TODO: This is not properly updating schedstats */
>>>> +	if (schedstat_enabled()) {
>>>> +		forced_migrations = schedstat_val(p->stats.nr_forced_migrations);
>>>> +		migrations_cold = schedstat_val(p->stats.nr_migrations_cold);
>>>> +		memset(&p->stats, 0, sizeof(p->stats));
>>>> +		schedstat_set(p->stats.nr_forced_migrations, forced_migrations);
>>>> +		schedstat_set(p->stats.nr_migrations_cold, migrations_cold);
>>>> +		schedstat_inc(p->stats.nr_migrations_cold);
>>>> +	}
>>>>  
>>>>  	trace_sched_move_numa(p, curr_cpu, target_cpu);
>>>>  	return stop_one_cpu(curr_cpu, migration_cpu_stop, &arg);
>>>> -- 
>>>> 2.27.0
>>>>
>>
>> Thanks,
>> Carlos
>>
> 

Thanks,
Carlos