Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <90041ab2-a8a1-8572-969c-143a707faed2@oracle.com>
Date:   Thu, 25 Aug 2022 02:13:17 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.12.0
Subject: Re: [PATCH 1/1] sched/fair: Fix inaccurate tally of ttwu_move_affine
Content-Language: en-US
To:     "Gautham R. Shenoy" <gautham.shenoy@amd.com>
Cc:     mingo@redhat.com, peterz@infradead.org, vincent.guittot@linaro.org,
        juri.lelli@redhat.com, dietmar.eggemann@arm.com, mgorman@suse.de,
        linux-kernel@vger.kernel.org,
        Daniel Jordan <daniel.m.jordan@oracle.com>
References: <20220810223313.386614-1-libo.chen@oracle.com>
 <Ywck+3E0DyNdUJRE@BLR-5CG11610CF.amd.com>
From:   Libo Chen <libo.chen@oracle.com>
In-Reply-To: <Ywck+3E0DyNdUJRE@BLR-5CG11610CF.amd.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UFBDTHU5ZHVoOGZaalVsZFpKNXpoT2plRm50OEpnbS8yQlYrNElsNC9WS3dY?=
 =?utf-8?B?SVJRSW9ONGRBd1V5UzNIaHhRZWhzWVVZeGFGSHNLYm5NOFBMWlNwQVBTdkt5?=
 =?utf-8?B?OXVtMGtuQkVNQnJIN1A3R1NMOEhLR1Vrd1BrcTNwUmI3UlpRWm4wTkY0dkp2?=
 =?utf-8?B?N2NVWEdycnZUK2dUNlB2RlJuMEtSSWRkbnlTSkdzODk1QlNqRkFIa1M0TzN5?=
 =?utf-8?B?TGRqbEQxVDgyQ3R3Szd3Yy84SjVDVXBzanowY3VxcDBXVW56blNQMjArWDhs?=
 =?utf-8?B?NEFZZk1WNFpXaitmOVJ0V3B2M1oycjk1VW43cTVxLzF5YU5PNVRYNUZFbEtY?=
 =?utf-8?B?L2lEWTd5b3l0WWhCWU0zZTg5ZTY4U1k2OWpnUTZpOFFGdHRscUZ5STVQclpM?=
 =?utf-8?B?T0l4dUJFRUpFK1hRZ2E2WWJrZ1hqLzJWengvZWwvZ3YzZjZQRnJQSUdGUVdQ?=
 =?utf-8?B?alNGcEIzRVNMOEhJY05QL0xzZDIzNnAyVDd1V2hDZklCOEJQWXpuL2tZMHdJ?=
 =?utf-8?B?eUt4b0JpM0NCY3JIajY3Wk1LNENCQ2hpQ2pzT3FtbDNjVDVJUzRxV3Q1bmpr?=
 =?utf-8?B?UUZKM296dURVU2hUcGJVeXd1OVFzVWVva3I4d1Bud2ZUb2VsL2VtWFZjVHJL?=
 =?utf-8?B?NEdqSXlKK1VzVU1qTmYxMVVqcnNRaCtxT2Z2MTZQa0RBVklmT0ZZR0lHSjhn?=
 =?utf-8?B?RFkyMS93cGc5R2Y3ODh0RlVQZksySDI4WlIzSlpHU08rT2hTZjVtbG1xWkFG?=
 =?utf-8?B?dFJra1RCZXIrMjVHUG5RYkFxVTBrSG52SjFvbW9ZYWlOQzJYU0NsRmMvMW1S?=
 =?utf-8?B?QzMrclV3SUxDRmFPTTNuc1ZGM0g4UGRIUmltMlFEWUsvY0FEZ2ZwdTlQZmdw?=
 =?utf-8?B?VWV2NkR0bXA1Q09HWUxmSDhMbUQ4SjU5UDVOeXIya3B3THIyVm9iUE45ZlYx?=
 =?utf-8?B?SUNKS2U3bEZ3ek0yVThqVzE5VGRHdW5ZaEorWGs2Z0JNanFjNnlnN2JQbEo4?=
 =?utf-8?B?Q0NPV2MzSVZ4c2tzWlNaRWgvY2FzZ3ZWbmxJZTFJaGllRkZ6dEhOTSsxZEhZ?=
 =?utf-8?B?WkpzOCtUTDEyT1ZHWHJVUkovNUJrMkJINXU3UFV3em1nWjdqWmUyVzIxVlB1?=
 =?utf-8?B?RkJ3RjFoc0oyQjVjMkZ4K3NpWUNpalJrcHRnUmh2K1BUem5ySERkaUM4QjV2?=
 =?utf-8?B?QzJSL1h1aU42dzBBR1dXbTNQVzU5dXNMOWVMQnNoZlpnSG5RV252VFBLUmd4?=
 =?utf-8?B?TVg2Q2FOSGtLOGZ6eHlMYmNmR2g0aVpqNXlIa1QybXBvbUxoU1MzY1U2aVlt?=
 =?utf-8?B?dUJqVDhmdnRaQm5SdXVrUCtadTN3QzRTclZBSENMcnF2UDBxbUJURmZpcXh5?=
 =?utf-8?B?UDB5ZmpJUkUrc3k2S3lMUXZuT3JkV0QxWDZnL2NhZHptdGZpdXNnbzVZaDIr?=
 =?utf-8?B?SHVNcXRsQjBudmlob1dqK3lvdVM0YSthdXB2TzRWUWdpcEFNdi8rVGQzR2ww?=
 =?utf-8?B?elRKWHpvTmVXaDV4T3A4Q2hQcHZQcVUxMXUwN21mQUplajFCUERGOWZOZVRK?=
 =?utf-8?B?dGhQYTNFc1ZxY2xtNnFTM295MjhYS0w4UWZDTHZwN09nVkNEcFpCclMzblYz?=
 =?utf-8?B?eWJYODNpMTNuOFB6WlhodG5HMWFaVHRCSkJwS1BNM0czVC9Nd29xR0EyMnRL?=
 =?utf-8?B?ckVrOUJFWXZzVVpGbmsvNk00bkx2c3NKRE84VU1ZV3JYcE5rYjNEVXdxcnZY?=
 =?utf-8?B?VXBiaGJ3M2d5aFRpK1pCd09IcGVneS9uQWZiTkUzTnJPVFoyWEtaRkZTaXhK?=
 =?utf-8?B?L25DVnNPK1REd3FYWHdoa1YzWFV6VG5HWEJwcnNqZkZHWGtTMGlBSmtQZXZX?=
 =?utf-8?B?TmMybWxOQ3I2Y0RodmRJVlRMcTkrbnZQQ3lpK0JaaXgydUlYSnBZSEdKdjcr?=
 =?utf-8?B?TmtCclBmM2kxSjNaQlA3QjUrR0Q4a2VzejJDUHlXVjdFSTlHRE9OS3hmeVFS?=
 =?utf-8?B?K1Y3a3dyd0NFWUQ4WXhIelJRK2lVNm1XR2JZZmdRNG1FSGViVUd2aWlLQmZv?=
 =?utf-8?B?eVFaOFZ6WTI2TmNPUXhGeUl2d0JnQ0lmcmd1S083dUNSU29GeUdZcU1ZK214?=
 =?utf-8?B?RDl4QVFyeGR5djFqdlp2ZWhnVVQrK09aNkRYYlFaMitzVUl0YjhBcjZWMTdo?=
 =?utf-8?B?UVo1Qm9qRnc4VlBCdGpkQzJBZkl4YmlIWUJJRlJLNWlVVjU3MEZrcEVlcnpK?=
 =?utf-8?B?NmVuMWV0dTBhZ2ZoQ21XVGJ6b2ZnPT0=?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 24474076-c8e3-4afa-ed42-08da867a0d3b
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4576.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2022 09:13:19.3604
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Py+iZaKdz1kd4ZQlwhiAaxR2u1nrvF5X/al6+bLnfwSrw57UDaHVaDTeATTj3m1T92GHd6FWJwWfehc7OhYadg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3923
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1
 definitions=2022-08-25_03,2022-08-22_02,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 suspectscore=0
 mlxlogscore=999 mlxscore=0 spamscore=0 adultscore=0 phishscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2207270000 definitions=main-2208250035
X-Proofpoint-GUID: E5_HWFcdPfXWJMHaaFkHtftT-QImUZfe
X-Proofpoint-ORIG-GUID: E5_HWFcdPfXWJMHaaFkHtftT-QImUZfe
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 8/25/22 00:30, Gautham R. Shenoy wrote:
> On Wed, Aug 10, 2022 at 03:33:13PM -0700, Libo Chen wrote:
>> There are scenarios where non-affine wakeups are incorrectly counted as
>> affine wakeups by schedstats.
>>
>> When wake_affine_idle() returns prev_cpu which doesn't equal to
>> nr_cpumask_bits, it will slip through the check: target == nr_cpumask_bits
>> in wake_affine() and be counted as if target == this_cpu in schedstats.
>>
>> Replace target == nr_cpumask_bits with target != this_cpu to make sure
>> affine wakeups are accurately tallied.
>>
>> Fixes: 806486c377e33 (sched/fair: Do not migrate if the prev_cpu is idle)
>> Suggested-by: Daniel Jordan <daniel.m.jordan@oracle.com>
>> Signed-off-by: Libo Chen <libo.chen@oracle.com>
>> ---
>>   kernel/sched/fair.c | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>> index da388657d5ac..b179da4f8105 100644
>> --- a/kernel/sched/fair.c
>> +++ b/kernel/sched/fair.c
>> @@ -6114,7 +6114,7 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p,
>>   		target = wake_affine_weight(sd, p, this_cpu, prev_cpu, sync);
>>   
>>   	schedstat_inc(p->stats.nr_wakeups_affine_attempts);
>> -	if (target == nr_cpumask_bits)
>> +	if (target != this_cpu)
>>   		return prev_cpu;
>
> This seems to be the right thing to do. However,..
>
> if this_cpu and prev_cpu were in the same LLC and we pick prev_cpu,
> technically is it still not an affine wakeup?
>
I think schedstats like ttwu_move_affine/ttwu_wake_remote is defined 
within a sched domain, so if the waking CPU and the previous CPU are in 
the same MC domain, then picking the previous CPU is a remote wakeup
within that MC. If the two candidate CPUs are from two different NUMA 
nodes, then picking the waking CPU is an affine wakeup within that NUMA 
domain. Correct me if I am wrong, this definition is consistent across
all levels of sched domains.

But I do understand that when two candidate CPUs are within an LLC,
     a) all the fast-path wakeups should be affine wakeups if your 
definition of an affine wakeup is a wakeup to the same LLC of the waker.
     b) select_idle_sibling() may pick a CPU in that LLC other than the 
two candidate CPUs which makes the affine/remote stats here useless even 
if we are consistent with ttwu_move_affine/ttwu_wake_remote
        definition.

I personally think it's just too much trouble to add additional code in 
the kernel to, let's say, treat all wakeups within an LLC as 
ttwu_move_affine. It's a lot easier to do that when you process 
schedstats data,
whether you want to treat all wakeups in LLC domains as affine wakeups 
or ignore ttwu_move_affine/ttwu_wake_remote stats from LLC domains.


>>   
>>   	schedstat_inc(sd->ttwu_move_affine);
>> -- 
>> 2.31.1
>>
> --
> Thanks and Regards
> gautham.