Received-SPF: pass (google.com: domain of linux-kernel+bounces-118319-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <f59c92ff-259b-5b89-9af5-fcaefccd4b23@amd.com>
Date: Tue, 26 Mar 2024 08:36:19 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.13.1
Subject: Re: [RFC PATCH 1/1] sched/eevdf: Skip eligibility check for current
 entity during wakeup preemption
Content-Language: en-US
To: Youssef Esmat <youssefesmat@chromium.org>
Cc: Ingo Molnar <mingo@redhat.com>, Peter Zijlstra <peterz@infradead.org>,
 Juri Lelli <juri.lelli@redhat.com>,
 Vincent Guittot <vincent.guittot@linaro.org>,
 Dietmar Eggemann <dietmar.eggemann@arm.com>,
 Steven Rostedt <rostedt@goodmis.org>, Ben Segall <bsegall@google.com>,
 Mel Gorman <mgorman@suse.de>, Daniel Bristot de Oliveira
 <bristot@redhat.com>, Valentin Schneider <vschneid@redhat.com>,
 linux-kernel@vger.kernel.org, Tobias Huschle <huschle@linux.ibm.com>,
 Luis Machado <luis.machado@arm.com>, Chen Yu <yu.c.chen@intel.com>,
 Abel Wu <wuyun.abel@bytedance.com>, Tianchen Ding
 <dtcccc@linux.alibaba.com>, Xuewen Yan <xuewen.yan94@gmail.com>,
 "Gautham R. Shenoy" <gautham.shenoy@amd.com>
References: <20240325060226.1540-1-kprateek.nayak@amd.com>
 <20240325060226.1540-2-kprateek.nayak@amd.com>
 <CA+q576OCK24VSp+s4OLD2ogO48i95y39_JO=zV=TwHSEg3_b1w@mail.gmail.com>
From: K Prateek Nayak <kprateek.nayak@amd.com>
In-Reply-To: <CA+q576OCK24VSp+s4OLD2ogO48i95y39_JO=zV=TwHSEg3_b1w@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?RThQU29ISjlOdHZtcW1pNHhwdG1Cc3ZuK0tERCtDd1VmVnMrWmZ0QzhOQ1RZ?=
 =?utf-8?B?OTE1WXpmd2NPNk16VmlBSFc5blRySHFBNnM4dG5LaVRYaEZMeVcyU2l2RVZM?=
 =?utf-8?B?bjBsYXNHcitVcnJQdVkxL2ZYblZTa2NqSlVqRUpNYStENXl5QmNrOHNjLy8w?=
 =?utf-8?B?Zi9BSzdaeDNuelo5ci8zZTg2UjZnNGl2b1k2RE1ESUV4NDJPdU5jQkZPSGdQ?=
 =?utf-8?B?eVlkUVg3K0VjSzVMSGxHOEE4TGpqQkh1OVMvZ1NiUEpCcjhLYUJJZGMxaHZ5?=
 =?utf-8?B?bGNPSTBKdy91amI4d3N5MWdURWk0aTZPYVN0bCttRjJWbEVZNXNIczZtRUQw?=
 =?utf-8?B?QzdZYzJLSElaaGdZZW4yRHgzajdLSGZCUXM2cnFhOExHYzV3TTFNTG5ERllH?=
 =?utf-8?B?UXF5RWUyS1RDQUMzSHFsaG05TFJMK2NqRHpyOWhDQmUxdnJ5SDhhQ2IwSWhH?=
 =?utf-8?B?SGxTU3pZTWN0VHRPeGpjNlc3U2xzaEptbW0yME52MUNVL1F2VVhJWDBCNW1I?=
 =?utf-8?B?TG5pa2hhdVlmWi8wUnRqaU9ubDI0c2NVZVZxbS9yNmVtcFdLTkxhN0lqeU1Q?=
 =?utf-8?B?V0c3eTl4eVphOGY5c1dXWEZCNFlET1RLU28ybElKS3lyZHY0QWtPQThOLzFP?=
 =?utf-8?B?U21ncTY5Nkxlclh0dzh1OUsyMkt0dHcveGhUbkpXcVkxK0Y5Vld1R1prRjJ6?=
 =?utf-8?B?YUhnWGYxb2l1TUkzMFZ2Vk1teWVUYlBvbGZjdjlrTVZxT0xGVGxzWFFuZUFF?=
 =?utf-8?B?RTJtOHU0MEp3UU5OR0xOdFNLTEE2cWptN3c1MncwbmgrSll3RVBjd25aeW9y?=
 =?utf-8?B?dlBKUHBZME0vdUFrMFgraTlwUzNmRzdSbFNUUGZ0Wi9mVUhDREN0b2I2TzRZ?=
 =?utf-8?B?UEZxemlOa0FLSzJYWWc4T2FMZjlacGxnMjBqMGwvYnFMUWR4WjFPaWV1aDl2?=
 =?utf-8?B?LzFCcWRIejJMQ25RME1pV3BFZWpjQ0RmajZOVkdvTG8xdUVtS1cyRjc1SEw0?=
 =?utf-8?B?K2VoZG5SaHJuR3hpWE9FWEdheEZtN3lBUFR3TzJ5WnRCQWFQNWthaFBkT3I4?=
 =?utf-8?B?b0h2MUJZVTVVKzFUc1BYRmt6MCtGVWJWWXpyRStFcncwa2dWMk5NemprUlBy?=
 =?utf-8?B?ZWV2MXRqQTZNczU2bHRBVE1VTDBHV0dDZFM2OWI0ckowZzBCRm1jSGdRN1dY?=
 =?utf-8?B?Ukd3VHZuUTRsV1ErMk9pMTBTK3ZtOWk4a09DbHFXalh5WGVYT3VkT2piMzda?=
 =?utf-8?B?aFlSMlBESkswRzB2MmtvZE91c2NDQzJNajZiMUk2WUk3TU9DMlVsdGJMdTRV?=
 =?utf-8?B?bit2V2tDMW9uWmpyb2Z2MVN1OUt6WGZTeGUybVZKVXpKVkFwUlU5dEJ0aFlz?=
 =?utf-8?B?OUdUZm10RXdIaWZpV1gxa2NidFRHamtUamJENFdFVnROWExtV2dmMFRTZjhK?=
 =?utf-8?B?NjFBeE1qQ004MUxCZXJWVzRUOE5jbFAvUmNvSDkvN1pZTkszdWlvd0JxY3lR?=
 =?utf-8?B?WFdrL3hnQkVrb1VFQzdnMExpQWpMSGt1eVllREpXV3FjT2FvN2RiYUtyOTNW?=
 =?utf-8?B?WE9sM2dmNVlPc0YyWUV3MUtkNDkvWDhvdlIvM09QaUNDQ2piOG1kbkZzWDZQ?=
 =?utf-8?B?THZCSEtQSUZhbHZHL0J1YTZrekVVVnBUZDQ3dGpmNDZpdktXbmJYajRvVVBo?=
 =?utf-8?B?Zi9hNVZEZ1JGTlVxS0tGV1ZtRkFxTGo5UmRZM05yWDNuYmFHZ0xiUEttUVZX?=
 =?utf-8?B?cHovKzAyVlExVDRnRkFyRjhsK1FzZU9GMWkvUW5zcmp0SkZVem9FVHBTaG52?=
 =?utf-8?B?MDIwQUVncElHcmZCRUYzUWlOb3BGNWlMSUJ6eFpzNmNKYmZnczZqVFRKRHQ4?=
 =?utf-8?B?bXFWeDIxcTNDMVpaZ0V2WStjNTRVYWJweDJvcWJSL3ByMm0wL2Vma05EeUc1?=
 =?utf-8?B?OGZWSHprZVlqVGJwM29McFBrQXhSdkx3dFQzYU0wN3ZoS1E3VDFVTmxMcC9C?=
 =?utf-8?B?d2Y1QlYwYkorSGpTa1NCdmZQcEtlZUlEMXhNQ21GaFdITVcvUHNTczZaQStJ?=
 =?utf-8?B?U01xZ3lEcTAwT1NHWHJqeFhRdnZNdEZlb29LTGR2ZVM2ZFR3Y3NFakxGNWxj?=
 =?utf-8?Q?JNBtsmMor9CCefOA7h1C7SHEG?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6a3cd61d-ac79-47d0-3d38-08dc4d41bd03
X-MS-Exchange-CrossTenant-AuthSource: MW2PR12MB2379.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Mar 2024 03:06:32.0458
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: b0e68Q8s1EIB85wIiFTw775gYvKyMaUfUJ2H6QbWAYXJy6ImkdToMU56aHJT+zkWY0cMBW7pMEFA/DIM3hzLzg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8873

Hello Youssef,

On 3/25/2024 8:43 PM, Youssef Esmat wrote:
> On Mon, Mar 25, 2024 at 1:03 AM K Prateek Nayak <kprateek.nayak@amd.com> wrote:
>>
>> With the curr entity's eligibility check, a wakeup preemption is very
>> likely when an entity with positive lag joins the runqueue pushing the
>> avg_vruntime of the runqueue backwards, making the vruntime of the
>> current entity ineligible. This leads to aggressive wakeup preemption
>> which was previously guarded by wakeup_granularity_ns in legacy CFS.
>> Below figure depicts one such aggressive preemption scenario with EEVDF
>> in DeathStarBench [1]:
>>
>>                                 deadline for Nginx
>>                    |
>>         +-------+  |                    |
>>     /-- | Nginx | -|------------------> |
>>     |   +-------+  |                    |
>>     |              |
>>     |   -----------|-------------------------------> vruntime timeline
>>     |              \--> rq->avg_vruntime
>>     |
>>     |   wakes service on the same runqueue since system is busy
>>     |
>>     |   +---------+|
>>     \-->| Service || (service has +ve lag pushes avg_vruntime backwards)
>>         +---------+|
>>           |        |
>>    wakeup |     +--|-----+                       |
>>  preempts \---->| N|ginx | --------------------> | {deadline for Nginx}
>>                 +--|-----+                       |
>>            (Nginx ineligible)
>>         -----------|-------------------------------> vruntime timeline
>>                    \--> rq->avg_vruntime
>>
>> When NGINX server is involuntarily switched out, it cannot accept any
>> incoming request, leading to longer turn around time for the clients and
>> thus loss in DeathStarBench throughput.
>>
>>     ==================================================================
>>     Test          : DeathStarBench
>>     Units         : Normalized latency
>>     Interpretation: Lower is better
>>     Statistic     : Mean
>>     ==================================================================
>>     tip         1.00
>>     eevdf       1.14 (+14.61%)
>>
>> For current running task, skip eligibility check in pick_eevdf() if it
>> has not exhausted the slice promised to it during selection despite the
>> situation having changed since. The behavior is guarded by
>> RUN_TO_PARITY_WAKEUP sched_feat to simplify testing. With
>> RUN_TO_PARITY_WAKEUP enabled, performance loss seen with DeathStarBench
>> since the merge of EEVDF disappears. Following are the results from
>> testing on a Dual Socket 3rd Generation EPYC server (2 x 64C/128T):
>>
>>     ==================================================================
>>     Test          : DeathStarBench
>>     Units         : Normalized throughput
>>     Interpretation: Higher is better
>>     Statistic     : Mean
>>     ==================================================================
>>     Pinning      scaling     tip            run-to-parity-wakeup(pct imp)
>>      1CCD           1       1.00            1.16 (%diff: 16%)
>>      2CCD           2       1.00            1.03 (%diff: 3%)
>>      4CCD           4       1.00            1.12 (%diff: 12%)
>>      8CCD           8       1.00            1.05 (%diff: 6%)
>>
>> With spec_rstack_overflow=off, the DeathStarBench performance with the
>> proposed solution is same as the performance on v6.5 release before
>> EEVDF was merged.
> 
> Thanks for sharing this Prateek.
> We actually noticed we could also gain performance by disabling
> eligibility checks (but disable it on all paths).
> The following are a few threads we had on the topic:
> 
> Discussion around eligibility:
> https://lore.kernel.org/lkml/CA+q576MS0-MV1Oy-eecvmYpvNT3tqxD8syzrpxQ-Zk310hvRbw@mail.gmail.com/
> Some of our results:
> https://lore.kernel.org/lkml/CA+q576Mov1jpdfZhPBoy_hiVh3xSWuJjXdP3nS4zfpqfOXtq7Q@mail.gmail.com/
> Sched feature to disable eligibility:
> https://lore.kernel.org/lkml/20231013030213.2472697-1-youssefesmat@chromium.org/
> 

Thank you for pointing me to the discussions. I'll give this a spin on
my machine and report back what I see. Hope some of it will help during
the OSPM discussion :)

>>
>> This may lead to newly waking task waiting longer for its turn on the
>> CPU, however, testing on the same system did not reveal any consistent
>> regressions with the standard benchmarks.
>>
>> Link: https://github.com/delimitrou/DeathStarBench/ [1]
>> Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
>> ---
>>  [..snip..]
>>
 
--
Thanks and Regards,
Prateek