Received-SPF: pass (google.com: domain of linux-kernel+bounces-100063-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Received-SPF: Pass (protection.outlook.com: domain of arm.com designates
 63.35.35.123 as permitted sender) receiver=protection.outlook.com;
 client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com;
 pr=C
Authentication-Results-Original: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=arm.com;
Message-ID: <cf813f92-9806-4449-b099-1bb2bd492b3c@arm.com>
Date: Tue, 12 Mar 2024 09:45:57 +0000
User-Agent: Mozilla Thunderbird
Subject: Re: EEVDF/vhost regression (bisected to 86bfbb7ce4f6 sched/fair: Add
 lag based placement)
To: "Michael S. Tsirkin" <mst@redhat.com>,
 Tobias Huschle <huschle@linux.ibm.com>
Cc: Jason Wang <jasowang@redhat.com>, Abel Wu <wuyun.abel@bytedance.com>,
 Peter Zijlstra <peterz@infradead.org>,
 Linux Kernel <linux-kernel@vger.kernel.org>, kvm@vger.kernel.org,
 virtualization@lists.linux.dev, netdev@vger.kernel.org, nd <nd@arm.com>
References: <42870.123121305373200110@us-mta-641.us.mimecast.lan>
 <20231213061719-mutt-send-email-mst@kernel.org>
 <25485.123121307454100283@us-mta-18.us.mimecast.lan>
 <20231213094854-mutt-send-email-mst@kernel.org>
 <20231214021328-mutt-send-email-mst@kernel.org>
 <92916.124010808133201076@us-mta-622.us.mimecast.lan>
 <20240121134311-mutt-send-email-mst@kernel.org>
 <07974.124020102385100135@us-mta-501.us.mimecast.lan>
 <20240201030341-mutt-send-email-mst@kernel.org>
 <89460.124020106474400877@us-mta-475.us.mimecast.lan>
 <20240311130446-mutt-send-email-mst@kernel.org>
Content-Language: en-US
From: Luis Machado <luis.machado@arm.com>
In-Reply-To: <20240311130446-mutt-send-email-mst@kernel.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
NoDisclaimer: true
X-Microsoft-Antispam-Message-Info-Original:
 F+mmDfTvH9FvVrcvJV3jWt6Zm67T4AFdXlDoWfjI7X3rZuJE3F/DjaXonnxzGrIM4NdFhQ7HCf535rZBO224nRtiK/V0DBaWVpeCAaf1OBWi7C487cq7NoUaEz3cIuVbQ1WJ3W7phETwZz6CdEq4LIxFGIT+mPgIENceKmxGy2cvJC44D4E+UoTWW4nOGtYsEWz850k/V4tKPT2nwW8FDJbM1srtzJnh5kAVePglJGob1xU3DkKZnk2Bmzvrt+fDL8A9BNILR1Png1J7v1uSGwvsLaMrvUeSIvECBuEqnErzkeiEujxd3nGMFx+6PnChfw/B795lmtMnG5xnQjq6E8pYmNT5J2ywxRwp591JZj/r6aT+YZtZw2XZpNUoorOyZOve5JIkfyeJEVJz8RQvDnqOQiURrzUWf6h4IOB/d6CdXKNsBOYOs3s73EaQ49ddsTggTZyDxfeYjv3qYyVK+FeLMIyC3gy/W7znzY+kLUOqLSbLrQhifypicItvYAzYTbI4/Xb1gWF2qDOZWTeC4k1M7P4SbmLGjkTI/jCOSvT+uInbLqlVNeyJDydwryaR1NoFxwSGwzrrGyFL28e3N7Yao7rnP31pc2ayXhamS4feT0ZjXv7khOZzEQ374J55
X-Forefront-Antispam-Report-Untrusted:
 CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR08MB3919.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(1800799015);DIR:OUT;SFP:1101;
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU2PR08MB10204
Original-Authentication-Results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=arm.com;
X-EOPAttributedMessage: 0
X-MS-Exchange-Transport-CrossTenantHeadersStripped:
 AM4PEPF00027A68.eurprd04.prod.outlook.com
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id-Prvs:
	4a7e6c97-7969-42c6-1517-08dc42793b26
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info:
	yqlid8dSjJg4WpqPj7iqbMe39HPh4O0pyiPMr7KjDblPRLzGnulfvTsaS0S4SMjOGaWPQ5ZbhBQGOFoahen5iCtXcrv7mqEIh4zBZuDTZlOZZxBLaQRQhjBnCmLBBlpNEf2X2wTlIcbG3a113/F5ab4dLB0gu29CyNCIfaC6IrVnOlxTe2sopTa+GkMppjJx9+simg0oVABqA8JCJvS7/T64XDRsxm+7fQ38sgMx+cvWBAeypzLL9Nuk9SUQxWYkZRoeGftSiLm1puCOHmguDiPDKKkjfq4x09DwBqvkCz+SeMERCgfXS0RfQrUkE8qy4K0OBV/Dn9aMyY2K/5Fxb7Rud1EDqUboqoDOVBPxwgk7GD523UdNBpAY4HO9TEX6/SGUYCXgP7nIRZIf/44Nl3kSCLaVmi62ajDVi6FCsN/e4peJ7c1J/cYU89zh0qtL9aSLamSo6ofwsOMZTauHjsSUrFVqFk8VBgvK+HJoe3ssbACrLE02DS/NicoODSErbkz9nd6P9cKk3LEWCF8DI/oRSOhBqeHROJJM3AhgwEEq70tjXrS+rm8e5h0U/gkIskH+7dtzjLgDhYLH0bXorpYe07LEfEU5dZkuYURujU+SLmWCaoA+H3/ypGTZx4nl0qyXkVWC+fmCdbhXz3pWheSGge+PV0qAhxPrfbGBEeWVatSOc6yUnA45WnicjfpuUvISsWcNov9j89tTZkts4g==
X-Forefront-Antispam-Report:
	CIP:63.35.35.123;CTRY:IE;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:64aa7808-outbound-1.mta.getcheckrecipient.com;PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com;CAT:NONE;SFS:(13230031)(82310400014)(36860700004)(1800799015)(376005);DIR:OUT;SFP:1101;
X-OriginatorOrg: arm.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Mar 2024 09:46:13.2174
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 5ccb1211-5ea4-4122-5db0-08dc42794142
X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d;Ip=[63.35.35.123];Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com]
X-MS-Exchange-CrossTenant-AuthSource:
	AM4PEPF00027A68.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR08MB9471

On 3/11/24 17:05, Michael S. Tsirkin wrote:
> On Thu, Feb 01, 2024 at 12:47:39PM +0100, Tobias Huschle wrote:
>> On Thu, Feb 01, 2024 at 03:08:07AM -0500, Michael S. Tsirkin wrote:
>>> On Thu, Feb 01, 2024 at 08:38:43AM +0100, Tobias Huschle wrote:
>>>> On Sun, Jan 21, 2024 at 01:44:32PM -0500, Michael S. Tsirkin wrote:
>>>>> On Mon, Jan 08, 2024 at 02:13:25PM +0100, Tobias Huschle wrote:
>>>>>> On Thu, Dec 14, 2023 at 02:14:59AM -0500, Michael S. Tsirkin wrote:
>>>>
>>>> -------- Summary --------
>>>>
>>>> In my (non-vhost experience) opinion the way to go would be either
>>>> replacing the cond_resched with a hard schedule or setting the
>>>> need_resched flag within vhost if the a data transfer was successfully
>>>> initiated. It will be necessary to check if this causes problems with
>>>> other workloads/benchmarks.
>>>
>>> Yes but conceptually I am still in the dark on whether the fact that
>>> periodically invoking cond_resched is no longer sufficient to be nice to
>>> others is a bug, or intentional.  So you feel it is intentional?
>>
>> I would assume that cond_resched is still a valid concept.
>> But, in this particular scenario we have the following problem:
>>
>> So far (with CFS) we had:
>> 1. vhost initiates data transfer
>> 2. kworker is woken up
>> 3. CFS gives priority to woken up task and schedules it
>> 4. kworker runs
>>
>> Now (with EEVDF) we have:
>> 0. In some cases, kworker has accumulated negative lag 
>> 1. vhost initiates data transfer
>> 2. kworker is woken up
>> -3a. EEVDF does not schedule kworker if it has negative lag
>> -4a. vhost continues running, kworker on same CPU starves
>> --
>> -3b. EEVDF schedules kworker if it has positive or no lag
>> -4b. kworker runs
>>
>> In the 3a/4a case, the kworker is given no chance to set the
>> necessary flag. The flag can only be set by another CPU now.
>> The schedule of the kworker was not caused by cond_resched, but
>> rather by the wakeup path of the scheduler.
>>
>> cond_resched works successfully once the load balancer (I suppose) 
>> decides to migrate the vhost off to another CPU. In that case, the
>> load balancer on another CPU sets that flag and we are good.
>> That then eventually allows the scheduler to pick kworker, but very
>> late.
> 
> Are we going anywhere with this btw?
> 
>

I think Tobias had a couple other threads related to this, with other potential fixes:

https://lore.kernel.org/lkml/20240228161018.14253-1-huschle@linux.ibm.com/

https://lore.kernel.org/lkml/20240228161023.14310-1-huschle@linux.ibm.com/