Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932179AbcDGUaT (ORCPT ); Thu, 7 Apr 2016 16:30:19 -0400 Received: from eu-smtp-delivery-143.mimecast.com ([146.101.78.143]:37058 "EHLO eu-smtp-delivery-143.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750993AbcDGUaQ convert rfc822-to-8bit (ORCPT ); Thu, 7 Apr 2016 16:30:16 -0400 Subject: Re: [PATCH RFC] sched/fair: let cpu's cfs_rq to reflect task migration To: Vincent Guittot References: <1459528717-17339-1-git-send-email-leo.yan@linaro.org> <20160401194948.GN3448@twins.programming.kicks-ass.net> <56FEF621.3070404@linaro.org> <20160402071154.GA7046@leoy-linaro> <20160404084821.GA18516@e105550-lin.cambridge.arm.com> <20160404183003.GA8697@intel.com> <20160405075112.GC18516@e105550-lin.cambridge.arm.com> <20160405001552.GB8697@intel.com> <5703EF38.2060204@arm.com> <20160406083702.GE18516@e105550-lin.cambridge.arm.com> <57055B41.6000906@arm.com> CC: Morten Rasmussen , Yuyang Du , Leo Yan , Steve Muckle , Peter Zijlstra , Ingo Molnar , linux-kernel , "eas-dev@lists.linaro.org" From: Dietmar Eggemann Message-ID: <5706C34B.3020508@arm.com> Date: Thu, 7 Apr 2016 21:30:03 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: X-Originating-IP: [2a02:c7d:8e16:a800:5cfd:9ae1:4981:b840] X-ClientProxiedBy: VI1PR05CA0023.eurprd05.prod.outlook.com (10.162.33.161) To DB5PR08MB1000.eurprd08.prod.outlook.com (10.166.14.9) X-MS-Office365-Filtering-Correlation-Id: fca4a88b-aa15-40a0-d215-08d35f2368ba X-Microsoft-Exchange-Diagnostics: 1;DB5PR08MB1000;2:Dpmdi+TOMyl4vCLXiP+BcPJSipihjV/h0Y2EOXbUC8W4/e4ODYBIqOMC52wUSJ3xhho93fqvj2n8mvpMwa/Usciek1AMwqlQtLWOtcZEEYinFlz3Kt/oAt3YynTMjdIpFFAEyLh9m6d4S5+XIVf0X2DeW+B4echUyuwX6qUkXNXFO/REBzrHtqQ6ZbXq5Gis;3:0zJ3vil2dpE8KnonSjVDzH4swnNVuyceAL6jC5YPOcUfc/eFTTaTj35rbM9U9/BWmRH6rtH01WngSLu42xx5/iFpfeTsGeYYkuweGHonvJ9fIiw1FCZJuYgOLYhIk0LH;25:0hHN+8KEm2itfP/OgRICBB5TEs/mdTPsqmaSkz2iZVy/X6eDE1oLEATjMc1gHIlT8pzddZUATQuoCCTc9K06JvPX0Gjn1Y3wwACHRswiJL9VoXvoNf9Lysb3EfYCLctXixN4ZY/ppcU+Y4Uc5YU23LydrKAqVy26Fy+QqhycgOL5+xw7GbJxTQYvxbTdO/Op6jAnIfwsHGOAt1FJCHzQkYh26kEaIAe2pG74OeUVM/EDoPH3EgMI/OryUiCJ/c3iJaQRT/Xh7q/ZHCDamLHRadM6+Qa9+ARg+2DREynQUBhXaPTM1xxr2grFA4XWvFmHh1ngfBBQ1+0UnBqrajgkyg==;20:1eEnFi//Me12w9qHwzCnun5HDWo4VKvqFHawZNXJ7szpU/XXe36Q1YgWrcMixY/p0rQkHHYCAqjlRHE6v2ekuAWmez+NJQaEtNKNnK3jPW5Sm8/4DAakFFCVjR/G0xkZtzx4MEJZ+Y07OgkSWDXHNu5CtaH4I7niiUti4XKCSfrShL4d1EpWH/+zFkAcp3kGZK0FRswwndDTrz0XC/YIEbElTwbVoWmXocJboYVKudJaMNqcSKV1vK/CGQ6GT5wG X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:DB5PR08MB1000; X-LD-Processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001);SRVR:DB5PR08MB1000;BCL:0;PCL:0;RULEID:;SRVR:DB5PR08MB1000; X-Microsoft-Exchange-Diagnostics: 1;DB5PR08MB1000;4:LJl5K306SoPVbTuAckmYzuMYc7HjOiiUAnp4Xnkulc+pgsrVTWJv4beu/HcwvRyEngLZp14kLG4h3tgqRkVCAry5iWTAAwMlSmpFslNeNnYLH0Ej//9jfgFYAeCyxMsxlJGGByJjJiUdU0tWie6MpLdmS3/SuBHQMNV+tKJhwV2wubQ4ox2Z0jc6daAEl7Ej8365ziYSuCu6MO9zHbQXNc6V1yfn6I2hTneHHvcuRULA9bZyJUWPZ3TUU3E2de34rkRV7UpWn7eCVkEWgiKXZv9OpI7/c2sefGEUBsHsPhzL6lf1J72UlZJvVq1HXLarw4GFE4QUGvrHxK7inemwsHSfmMus9NB2AIh1vywXo+YPuttp+hBC+J3PGErUPoXT X-Forefront-PRVS: 0905A6B2C7 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(377454003)(40434004)(24454002)(59896002)(4326007)(5008740100001)(4001350100001)(5004730100002)(64126003)(42186005)(77096005)(5890100001)(36756003)(81166005)(19580395003)(83506001)(80316001)(19580405001)(586003)(6116002)(2950100001)(76176999)(1096002)(65806001)(47776003)(86362001)(33656002)(65956001)(2906002)(65816999)(110136002)(54356999)(87266999)(50986999)(189998001)(23676002)(561944003)(92566002)(50466002)(3826002);DIR:OUT;SFP:1101;SCL:1;SRVR:DB5PR08MB1000;H:[IPv6:2a02:c7d:8e16:a800:5cfd:9ae1:4981:b840];FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtEQjVQUjA4TUIxMDAwOzIzOmRSamRnUUhUL2U3YTN1ODRZVlJybERjanZt?= =?utf-8?B?eVhrZmp3WE5VK0NPUnFSbUVwUHFucFllSG5XUUJ2R2oyQWE3MytzN21aOWFZ?= =?utf-8?B?SDlxWkIyd3hRVjU3NW4yV1BhMjJUZzcyVXdPem1FUHpqMDZrZUVvTWxGTXBL?= =?utf-8?B?VnQvTEpOS0NWTzdmNDVMK1B2c05va2lZSlhhWTJhenc1THBMZTJZb1kxQ1Mz?= =?utf-8?B?UU01dTlkWTBZcVU0SG95cnJiMnBpblJyTHoxTzVxemxFbjRRL3l1TnQydDdv?= =?utf-8?B?TmZOM0pqRlc2WjVGajlDZGVQRHhuKzl4UkpkdytCSm1MdUZ6TTZHMlFJWi8w?= =?utf-8?B?YU45VnFEM3Bwa0VXWjlscTlDZTQvVmxKVFRrYUlZOUw4N2lkWmJLMzAyTzVP?= =?utf-8?B?MyticFB3YWt1LytiS0xVUklBM3dsOW1MMWdFZTA5akROS1R3bVBkd1p0QUJ0?= =?utf-8?B?TTF4Nk1HVDV0WDdMcTdIRkJFN1k1NTRoTS9rVFJaQlUxRGtzZ2JkOG5nMnBi?= =?utf-8?B?d2cvam1mbm45aW11ekxYMXBkcHdMWUo2NmVSUzVrVXExenZ3Q0xNaTdVOFJz?= =?utf-8?B?d21sMWY1M1dGWUhmTUhyeWVNK2RqbWczNHNkaFNjME1PejJEakxPMDdMb0dW?= =?utf-8?B?Y1dMV2FMelFXY05IU3RWY24wR0hEVERhbXp1bVNmbWVHYnRqYmJUM3hMWVJT?= =?utf-8?B?SzhTcnhUQ3lSM2hoY21LNWdsbjY2OWFXUFBaQ3BPZHJtak5nbW1IWnNVMVpt?= =?utf-8?B?anhhMFRjZWMwY3EwLzNXUEpkbVhWM294ZXRUNTVpUzVYVWx4OUFMTmtYTlpJ?= =?utf-8?B?NGxzZU03K1MxNWhvSnNzQmowRW5hbGxTSElLaVNsVTJUSXVjMlV0R1dUUkRT?= =?utf-8?B?TUM5RjNJUjA2NXdLcnJ2TlI2cmZ6eCtxL2FFUFVlbzBUemo2emh6bVVyMnJT?= =?utf-8?B?Qm93dzV4VlpnZ0h6OUlrZzl0N1VJbFRLRW5MNGdjWUNoL3o5QUxiNW92WFFs?= =?utf-8?B?TElQR3FwYnp5OHUyb2NyUFZHRGVFTjR5M0l3VE5vdmRVTHJCeEtaN09kb3Vp?= =?utf-8?B?RlRCek1nbzlaTEp1ZHZYdG9jWVlMNmVsV1JkcjZIa1RKNC81bTZFRzhWRS9z?= =?utf-8?B?UUhoWHRwN1kzS3VSN2lpNFVOaHZKeVJoeUs2MmRzRWc5ZWMxdmhKUkZ0MFFI?= =?utf-8?B?SUtGZ256RE9HemxLaFJxN0llZS9QTzl1VVNXSEs2ZnhjMnRIUEFuN25JRXFw?= =?utf-8?B?TDYxSTh3cnZSanN2Sk5pVHhzcXRNMW9oaU5lNlczQUpwRkFYYklBNSs3NUtS?= =?utf-8?B?Tk5YR1N0bk5yK2wxVnBNQkdUclJYdTl3VCt0MXdZcTVTZ3JUMTB3RUtDbUVk?= =?utf-8?B?aXBuM1lMYW8xS1FkcjJaM3pjUE1naW5EYzFYa3ErZnl3K3ladXUra2xmazlU?= =?utf-8?B?dmtVVWlzOTFsTCtFa2l6VE9QWlVlejN6Tnh1UkdXdkFRbkVlc1dvdHJvdjFu?= =?utf-8?Q?UR4Wb6wmOqP5MYC3PhrLzPE9o=3D?= X-Microsoft-Exchange-Diagnostics: 1;DB5PR08MB1000;5:ndSPhc8EbBBXmVeYB3N+3wbu8GDeJ3I3KcM0k4OH99VaLVNX6EswuThvH2mTohyBE5LsUNWthfX/v+QspOBOFXDlb2EjJo5wum19HWGxR+geRgMF9mOGg7SmHCbk53nWB5tqqTd/vz53TbBCL72Xvg==;24:C3CVwFrvMdq5WzlkZ9lzAsFmwni5I6OjSJlFCracjNikFMR3SBTJMoj1NOSZie0YdtycK66peP6EDKceJwR/Zzig+orG16M88eAOmE5PVu0=;20:k+4zj5+AWzuWUDWoIz7UPdwCS0DF+3eWVWMZHVfwb88s7/XPr3wUhHoIBLA4CfmsJ9yBgLAG7QTuUTav5m4kRNCdlQFVX68mS9ZKsb2IJSSxUInAFTG+rvMpKx5F5UA/EVQkCqnlk0w6b4if9y3rjkcfiGCOM66xtXpj9LGFmiw= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Apr 2016 20:30:06.5252 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5PR08MB1000 X-MC-Unique: yRCCm8qwRwmB_qX0UyFYpA-1 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3072 Lines: 70 Hi Vincent, On 04/07/2016 02:04 PM, Vincent Guittot wrote: > Hi Dietmar, > > On 6 April 2016 at 20:53, Dietmar Eggemann wrote: >> On 06/04/16 09:37, Morten Rasmussen wrote: >>> On Tue, Apr 05, 2016 at 06:00:40PM +0100, Dietmar Eggemann wrote: [...] >> @@ -2910,8 +2920,13 @@ static void attach_entity_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *s >> if (!entity_is_task(se)) >> return; >> >> - rq_of(cfs_rq)->cfs.avg.util_avg += se->avg.util_avg; >> - rq_of(cfs_rq)->cfs.avg.util_sum += se->avg.util_sum; >> + if (&rq_of(cfs_rq)->cfs == cfs_rq) { >> + rq_of(cfs_rq)->cfs.avg.util_avg += se->avg.util_avg; >> + rq_of(cfs_rq)->cfs.avg.util_sum += se->avg.util_sum; >> + } else { >> + rq_of(cfs_rq)->cfs.added_util_avg = se->avg.util_avg; >> + rq_of(cfs_rq)->cfs.added_util_sum = se->avg.util_sum; >> + } >> } > > Don't you also need similar thing for the detach ? Maybe? I ran workloads in tg's and checked last_update_time of cfs_rq and &rq_of(cfs_rq)->cfs and they always were in sync. That's obviously only the call-stack 'task_move_group_fair() -> detach_task_cfs_rq() -> detach_entity_load_avg()' and not the one starting from switched_from_fair(). [...] >> But attach_entity_load_avg() is not only called in enqueue_entity_load_avg() for migrated >> tasks but also in attach_task_cfs_rq() which is called from switched_to_fair() and >> task_move_group_fair() where we can't assume that after the enqueue_entity_load_avg() a >> call to update_cfs_rq_load_avg() follows like in >> >> enqueue_task_fair(): >> >> for_each_sched_entity(se) >> enqueue_entity() >> enqueue_entity_load_avg() >> update_cfs_rq_load_avg(now, cfs_rq) >> if (migrated) attach_entity_load_avg() >> >> for_each_sched_entity(se) >> update_load_avg() >> update_cfs_rq_load_avg(now, cfs_rq) >> >> >> Not sure if we can just update the root cfs_rq to se->avg.last_update_time before we add >> se->avg.util_[avg/sum] to rq_of(cfs_rq)->cfs.avg.util_[avg/sum] in attach_entity_load_avg()? >> >> cfs_rq throttling has to be considered as well ... > > IIUC this new proposal, the utilization of a task will be accounted on > the utilization of the root cfs_rq thanks to > tsk->se->cfs_rq->tg->se[cpu]->... down to the root cfs_rq. Then, you > directly add the utilization of the newly enqueued task in the root > cfs_rq. Not sure if you're referring to this, but in __update_load_avg() I suppress the utilization update for se's w/ !entity_is_task(se) and cfs_rq's w/ &rq_of(cfs_rq)->cfs != cfs_rq so preventing the first case. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.