Received-SPF: pass (google.com: domain of linux-kernel+bounces-104164-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <65f40816.050a0220.372fc.7d58SMTPIN_ADDED_BROKEN@mx.google.com>
Date: Fri, 15 Mar 2024 09:33:49 +0100
From: Tobias Huschle <huschle@linux.ibm.com>
To: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Luis Machado <luis.machado@arm.com>, Jason Wang <jasowang@redhat.com>,
        Abel Wu <wuyun.abel@bytedance.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Linux Kernel <linux-kernel@vger.kernel.org>, kvm@vger.kernel.org,
        virtualization@lists.linux.dev, netdev@vger.kernel.org,
        nd <nd@arm.com>
Subject: Re: EEVDF/vhost regression (bisected to 86bfbb7ce4f6 sched/fair: Add
 lag based placement)
References: <20231214021328-mutt-send-email-mst@kernel.org>
 <92916.124010808133201076@us-mta-622.us.mimecast.lan>
 <20240121134311-mutt-send-email-mst@kernel.org>
 <07974.124020102385100135@us-mta-501.us.mimecast.lan>
 <20240201030341-mutt-send-email-mst@kernel.org>
 <89460.124020106474400877@us-mta-475.us.mimecast.lan>
 <20240311130446-mutt-send-email-mst@kernel.org>
 <cf813f92-9806-4449-b099-1bb2bd492b3c@arm.com>
 <73123.124031407552500165@us-mta-156.us.mimecast.lan>
 <20240314110649-mutt-send-email-mst@kernel.org>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240314110649-mutt-send-email-mst@kernel.org>

On Thu, Mar 14, 2024 at 11:09:25AM -0400, Michael S. Tsirkin wrote:
> 
> Thanks a lot! To clarify it is not that I am opposed to changing vhost.
> I would like however for some documentation to exist saying that if you
> do abc then call API xyz. Then I hope we can feel a bit safer that
> future scheduler changes will not break vhost (though as usual, nothing
> is for sure).  Right now we are going by the documentation and that says
> cond_resched so we do that.
> 
> -- 
> MST
> 

Here I'd like to add that we have two different problems:

1. cond_resched not working as expected
   This appears to me to be a bug in the scheduler where it lets the cgroup, 
   which the vhost is running in, loop endlessly. In EEVDF terms, the cgroup
   is allowed to surpass its own deadline without consequences. One of my RFCs
   mentioned above adresses this issue (not happy yet with the implementation).
   This issue only appears in that specific scenario, so it's not a general 
   issue, rather a corner case.
   But, this fix will still allow the vhost to reach its deadline, which is
   one full time slice. This brings down the max delays from 300+ms to whatever
   the timeslice is. This is not enough to fix the regression.

2. vhost relying on kworker being scheduled on wake up
   This is the bigger issue for the regression. There are rare cases, where
   the vhost runs only for a very short amount of time before it wakes up 
   the kworker. Simultaneously, the kworker takes longer than usual to 
   complete its work and takes longer than the vhost did before. We
   are talking 4digit to low 5digit nanosecond values.
   With those two being the only tasks on the CPU, the scheduler now assumes
   that the kworker wants to unfairly consume more than the vhost and denies
   it being scheduled on wakeup.
   In the regular cases, the kworker is faster than the vhost, so the 
   scheduler assumes that the kworker needs help, which benefits the
   scenario we are looking at.
   In the bad case, this means unfortunately, that cond_resched cannot work
   as good as before, for this particular case!
   So, let's assume that problem 1 from above is fixed. It will take one 
   full time slice to get the need_resched flag set by the scheduler
   because vhost surpasses its deadline. Before, the scheduler cannot know
   that the kworker should actually run. The kworker itself is unable
   to communicate that by itself since it's not getting scheduled and there 
   is no external entity that could intervene.
   Hence my argumentation that cond_resched still works as expected. The
   crucial part is that the wake up behavior has changed which is why I'm 
   a bit reluctant to propose a documentation change on cond_resched.
   I could see proposing a doc change, that cond_resched should not be
   used if a task heavily relies on a woken up task being scheduled.