Subject: Re: [patch 0/3] KVM KVM_HC_RT_PRIO hypercall support
To: Marcelo Tosatti <mtosatti@redhat.com>
Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org
References: <20170921113835.031375194@redhat.com>
 <0e9df6b6-f8ea-ad55-3308-9e583128cf46@siemens.com>
 <20170922011857.GC20133@amt.cnet>
From: Jan Kiszka <jan.kiszka@siemens.com>
Message-ID: <1663b883-a59e-2093-5ccb-308cc7f0bda5@siemens.com>
Date: Fri, 22 Sep 2017 08:23:02 +0200
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12)
 Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666
MIME-Version: 1.0
In-Reply-To: <20170922011857.GC20133@amt.cnet>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 4040
Lines: 90

On 2017-09-22 03:19, Marcelo Tosatti wrote:
> On Thu, Sep 21, 2017 at 07:45:32PM +0200, Jan Kiszka wrote:
>> On 2017-09-21 13:38, Marcelo Tosatti wrote:
>>> When executing guest vcpu-0 with FIFO:1 priority, which is necessary to
>>> deal with the following situation:
>>>
>>> VCPU-0 (housekeeping VCPU)              VCPU-1 (realtime VCPU)
>>>
>>> raw_spin_lock(A)
>>> interrupted, schedule task T-1          raw_spin_lock(A) (spin)
>>>
>>> raw_spin_unlock(A)
>>>
>>> Certain operations must interrupt guest vcpu-0 (see trace below).
>>>
>>> To fix this issue, only change guest vcpu-0 to FIFO priority
>>> on spinlock critical sections (see patch).
>>>
>>> Hang trace
>>> ==========
>>>
>>> Without FIFO priority:
>>>
>>> qemu-kvm-6705  [002] ....1.. 767785.648964: kvm_exit: reason IO_INSTRUCTION rip 0xe8fe info 1f00039 0
>>> qemu-kvm-6705  [002] ....1.. 767785.648965: kvm_exit: reason IO_INSTRUCTION rip 0xe911 info 3f60008 0
>>> qemu-kvm-6705  [002] ....1.. 767785.648968: kvm_exit: reason IO_INSTRUCTION rip 0x8984 info 608000b 0
>>> qemu-kvm-6705  [002] ....1.. 767785.648971: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
>>> qemu-kvm-6705  [002] ....1.. 767785.648974: kvm_exit: reason IO_INSTRUCTION rip 0xb514 info 3f60000 0
>>> qemu-kvm-6705  [002] ....1.. 767785.648977: kvm_exit: reason PENDING_INTERRUPT rip 0x8052 info 0 0
>>> qemu-kvm-6705  [002] ....1.. 767785.648980: kvm_exit: reason IO_INSTRUCTION rip 0xeee6 info 200040 0
>>> qemu-kvm-6705  [002] ....1.. 767785.648999: kvm_exit: reason EPT_MISCONFIG rip 0x2120 info 0 0
>>>
>>> With FIFO priority:
>>>
>>> qemu-kvm-7636  [002] ....1.. 768218.205065: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
>>> qemu-kvm-7636  [002] ....1.. 768218.205068: kvm_exit: reason IO_INSTRUCTION rip 0x8984 info 608000b 0
>>> qemu-kvm-7636  [002] ....1.. 768218.205071: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
>>> qemu-kvm-7636  [002] ....1.. 768218.205074: kvm_exit: reason IO_INSTRUCTION rip 0x8984 info 608000b 0
>>> qemu-kvm-7636  [002] ....1.. 768218.205077: kvm_exit: reason IO_INSTRUCTION rip 0xb313 info 1f70008 0
>>> ..
>>>
>>> Performance numbers (kernel compilation with make -j2)
>>> ======================================================
>>>
>>> With hypercall: 4:40.  (make -j2)
>>> Without hypercall: 3:38.  (make -j2)
>>>
>>> Note for NFV workloads spinlock performance is not relevant
>>> since DPDK should not enter the kernel (and housekeeping vcpu
>>> performance is far from a key factor).
>>>
>>> Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>>>
>>
>> That sounds familiar, though not yet the same: :)
>>
>> http://git.kiszka.org/?p=linux-kvm.git;a=shortlog;h=refs/heads/queues/paravirt-sched
>> (paper: http://lwn.net/images/conf/rtlws11/papers/proc/p18.pdf)
>>
>> I suppose your goal is not to enable the host to follow the guest
>> scheduler priority completely but only to have prio-ceiling for such
>> short critical sections. Maybe still useful to think ahead about future
>> extensions when actually introducing such an interface.
> 
> Hi Jan!
> 
> Hum... I'll take a look at your interface/paper and get back to you.
> 
>> But shouldn't there be some limits on the maximum prio the guest can select?
> 
> The SCHED_FIFO prio is fixed, selectable when QEMU starts. Do you
> envision any other use case than a fixed priority value selectable
> at QEMU initialization?

Oh, indeed, this is a pure prio-ceiling variant with host-defined
ceiling value.

But it's very inefficient to use a hypercall for entering and leaving
each and every sections. I would strongly recommend using a lazy scheme
where the guest writes the desired state into a shared memory page, and
the host only evaluates that prior to taking a scheduling decision, or
at least only on real vmexits. We're using such scheme successfully to
accelerate the fast path of prio-ceiling for pthread mutexes in the
Xenomai real-time extension.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux