Received-SPF: pass (google.com: domain of linux-kernel+bounces-106247-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Received-SPF: Pass (protection.outlook.com: domain of arm.com designates
 63.35.35.123 as permitted sender) receiver=protection.outlook.com;
 client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com;
 pr=C
Authentication-Results-Original: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=arm.com;
Message-ID: <66c4286e-deaf-44a0-be62-0928529ae73f@arm.com>
Date: Mon, 18 Mar 2024 14:45:27 +0000
User-Agent: Mozilla Thunderbird
Subject: Re: [RFC] sched/eevdf: sched feature to dismiss lag on wakeup
To: Tobias Huschle <huschle@linux.ibm.com>
Cc: linux-kernel@vger.kernel.org, mingo@redhat.com, peterz@infradead.org,
 juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com,
 rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de,
 bristot@redhat.com, vschneid@redhat.com, sshegde@linux.vnet.ibm.com,
 srikar@linux.vnet.ibm.com, linuxppc-dev@lists.ozlabs.org, nd <nd@arm.com>
References: <20240228161018.14253-1-huschle@linux.ibm.com>
 <5a32e8e1-67cf-4296-a655-f0fc35dc880a@arm.com>
 <ZfL/hROYxQudcTuX@DESKTOP-2CCOB1S.>
Content-Language: en-US
From: Luis Machado <luis.machado@arm.com>
In-Reply-To: <ZfL/hROYxQudcTuX@DESKTOP-2CCOB1S.>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
NoDisclaimer: true
X-Microsoft-Antispam-Message-Info-Original:
 7aoamcJB01kBge1l5H6ivJJw6MqkPuljgJG9ERuLGNTjdRG8j6naeNaFUCAp2m01M6ThYIiyl5A5/W6McB5EcG9hzoUlKe3dhP1yH5g0BJetNkbzVd/zBW+huNX86cncf5QgaXzWwpGz/T1aBRPJiZe79bbvKrD7KLdxg1s04M1o2klGGyXf6Y9zRJtedzVI0TVojxmAuCxjh4cd0iylSkut9P3qTgb6nr28W3k36XWXOOVaHiDgmMsd2FGmD2drtY0MFo7JrAJbrpo0j4cFPamn+uWtKZAccgUZ8zrKPtvSmvz4LYXpZi93uQ7lhiPCgTp8beylqFaF+QfhJMH17opGIMrZVEENNKhZ2AZJxPp0cXTmsHoQihi5K8hLzB4j6H/Jxfz2Sfynr2jLl4YBtKKW/3lAW81TnyTzQNACL/jZUDl+Q8qx7NlkW/Bi/HOEjfc38yQja8G8ZBl5F4EHx/NdZsFs84uuz75buTeFuskr6ShbT15Ne2BdoiGPezThYmMRZm9bwqxQjov1WTAo7LQ3LkL63mhKouZeRzYbgkb88ecrP8g7tKWmGCvzEHc4L6igUnb9stW67EUosEWShoP88LiqclwZu3rTGaJ9cbU=
X-Forefront-Antispam-Report-Untrusted:
 CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR08MB3919.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(7416005)(366007)(376005);DIR:OUT;SFP:1101;
X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV1PR08MB10536
Original-Authentication-Results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=arm.com;
X-EOPAttributedMessage: 0
X-MS-Exchange-Transport-CrossTenantHeadersStripped:
 DB1PEPF000509F6.eurprd02.prod.outlook.com
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id-Prvs:
	255463f8-ab7e-4d72-a5af-08dc475a0f8b
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	ypELNA7PL9E3vSS/f18nB0zm48+IGp83iH3/fXcHmOTfjB/jgBFVYkW0YJhOutiUVp8pM3MuIRE0gdv8RNk/0mIBmjZEQeJZVLvrXqO7shJg94aWAtC02g7PtGjdmNyF91nqTALHgQcHOGHS7/M6aHv35xaQUoE5KW32a+ZpplnajP8xJHJ9ydI76alRfSXnzwSxrb42UtID/HCP/eG7kjtPVPg5Aav6SePnZ2aTFsVzimO82roo3UZajlvWgY2egZAb5cAlzTnxZtI6jtgKhzJQC2pya9zUNXWCm0SZLsVu+Y/czsKKvuCWfIG7x4l4PYcYqS/1+SVo/kIpWHp67t1Caq2ctgFVmhC5Q99NS90FX4gsdhPB0d3qR9uqbfdFlvtBrbltr5NbTFFvphO4QQu1LM9GEPuDyAQowOVWFJ0XfzFFNW9Sin8Vg0Qfm507nsaRCzXjIY13uSxE/qdrVo63QCjDdLDvzXfIotrosgV5cVT5AKySuW+siEkJbm4F/RixgpMWhwwkauVo1xnyDNyLRaQHhKka5+UuBzGvz8gHbH7JMrsbfjDDVBsMfA137rP7pRbCaGonjh4d1kf0eCGc7CPmoWsGtMuFbLinK+41spAocLrwqUTk8cPncxjwEDKnh0ujQPCeEBwDNUUqzURrzJwf3eDIc9QEltTVWsc4CtaeroaupLrKVI8LpQBjUrrn3CSd4cNBYCMO5Gp1jQ==
X-Forefront-Antispam-Report:
	CIP:63.35.35.123;CTRY:IE;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:64aa7808-outbound-1.mta.getcheckrecipient.com;PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com;CAT:NONE;SFS:(13230031)(82310400014)(376005)(36860700004)(1800799015);DIR:OUT;SFP:1101;
X-OriginatorOrg: arm.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Mar 2024 14:45:49.9439
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: d4b5002b-572d-407b-5406-08dc475a1aa5
X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d;Ip=[63.35.35.123];Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com]
X-MS-Exchange-CrossTenant-AuthSource:
	DB1PEPF000509F6.eurprd02.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR08MB8412

On 3/14/24 13:45, Tobias Huschle wrote:
> On Fri, Mar 08, 2024 at 03:11:38PM +0000, Luis Machado wrote:
>> On 2/28/24 16:10, Tobias Huschle wrote:
>>>
>>> Questions:
>>> 1. The kworker getting its negative lag occurs in the following scenario
>>>    - kworker and a cgroup are supposed to execute on the same CPU
>>>    - one task within the cgroup is executing and wakes up the kworker
>>>    - kworker with 0 lag, gets picked immediately and finishes its
>>>      execution within ~5000ns
>>>    - on dequeue, kworker gets assigned a negative lag
>>>    Is this expected behavior? With this short execution time, I would
>>>    expect the kworker to be fine.
>>
>> That strikes me as a bit odd as well. Have you been able to determine how a negative lag
>> is assigned to the kworker after such a short runtime?
>>
> 
> I did some more trace reading though and found something.
> 
> What I observed if everything runs regularly:
> - vhost and kworker run alternating on the same CPU
> - if the kworker is done, it leaves the runqueue
> - vhost wakes up the kworker if it needs it
> --> this means:
>   - vhost starts alone on an otherwise empty runqueue
>   - it seems like it never gets dequeued
>     (unless another unrelated task joins or migration hits)
>   - if vhost wakes up the kworker, the kworker gets selected
>   - vhost runtime > kworker runtime 
>     --> kworker gets positive lag and gets selected immediately next time
> 
> What happens if it does go wrong:
> From what I gather, there seem to be occasions where the vhost either
> executes suprisingly quick, or the kworker surprinsingly slow. If these
> outliers reach critical values, it can happen, that
>    vhost runtime < kworker runtime
> which now causes the kworker to get the negative lag.
> 
> In this case it seems like, that the vhost is very fast in waking up
> the kworker. And coincidentally, the kworker takes, more time than usual
> to finish. We speak of 4-digit to low 5-digit nanoseconds.
> 
> So, for these outliers, the scheduler extrapolates that the kworker 
> out-consumes the vhost and should be slowed down, although in the majority
> of other cases this does not happen.

Thanks for providing the above details Tobias. It does seem like EEVDF is strict
about the eligibility checks and making tasks wait when their lags are negative, even
if just a little bit as in the case of the kworker.

There was a patch to disable the eligibility checks (https://lore.kernel.org/lkml/20231013030213.2472697-1-youssefesmat@chromium.org/),
which would make EEVDF more like EVDF, though the deadline comparison would
probably still favor the vhost task instead of the kworker with the negative lag.

I'm not sure if you tried it, but I thought I'd mention it.