Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Sun, 26 Sep 2021 10:02:28 +0100
Message-ID: <877df3btgb.wl-maz@kernel.org>
From:   Marc Zyngier <maz@kernel.org>
To:     Paolo Bonzini <pbonzini@redhat.com>
Cc:     Sean Christopherson <seanjc@google.com>,
        Huacai Chen <chenhuacai@kernel.org>,
        Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>,
        Paul Mackerras <paulus@ozlabs.org>,
        Christian Borntraeger <borntraeger@de.ibm.com>,
        Janosch Frank <frankja@linux.ibm.com>,
        James Morse <james.morse@arm.com>,
        Alexandru Elisei <alexandru.elisei@arm.com>,
        Suzuki K Poulose <suzuki.poulose@arm.com>,
        David Hildenbrand <david@redhat.com>,
        Cornelia Huck <cohuck@redhat.com>,
        Claudio Imbrenda <imbrenda@linux.ibm.com>,
        Vitaly Kuznetsov <vkuznets@redhat.com>,
        Wanpeng Li <wanpengli@tencent.com>,
        Jim Mattson <jmattson@google.com>,
        Joerg Roedel <joro@8bytes.org>,
        linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu,
        linux-mips@vger.kernel.org, kvm@vger.kernel.org,
        kvm-ppc@vger.kernel.org, linux-kernel@vger.kernel.org,
        David Matlack <dmatlack@google.com>,
        Jing Zhang <jingzhangos@google.com>
Subject: Re: [PATCH 07/14] KVM: Don't block+unblock when halt-polling is successful
In-Reply-To: <80d90ee6-0d43-3735-5c26-be8c3d72d493@redhat.com>
References: <20210925005528.1145584-1-seanjc@google.com>
        <20210925005528.1145584-8-seanjc@google.com>
        <878rzlass2.wl-maz@kernel.org>
        <80d90ee6-0d43-3735-5c26-be8c3d72d493@redhat.com>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1
 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk

On Sun, 26 Sep 2021 07:27:28 +0100,
Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> On 25/09/21 11:50, Marc Zyngier wrote:
> >> there is no need for arm64 to put/load
> >> the vGIC as KVM hasn't relinquished control of the vCPU in any way.
> > 
> > This doesn't mean that there is no requirement for any state
> > change. The put/load on GICv4 is crucial for performance, and the VMCR
> > resync is a correctness requirement.
> 
> I wouldn't even say it's crucial for performance: halt polling cannot
> work and is a waste of time without (the current implementation of)
> put/load.

Not quite. A non-V{LPI,SGI} could still be used as the a wake-up from
WFI (which is the only reason we end-up on this path). Only LPIs (and
SGIs on GICv4.1) can be directly injected, meaning that SPIs and PPIs
still follow the standard SW injection model.

However, there is still the ICH_VMCR_EL2 requirement (to get the
up-to-date priority mask and group enable bits) for SW-injected
interrupt wake-up to work correctly, and I really don't want to save
that one eagerly on each shallow exit.

> However, is activating the doorbell necessary?  If possible, polling
> the VGIC directly for pending VLPIs without touching the ITS (for
> example by emulating IAR reads) may make sense.  IIUC that must be
> done at EL2 though, so maybe it would even make sense to move all of
> halt polling to EL2 for the nVHE case.  It all depends on benchmark
> results, of course.

No, there is no architectural way to observe the VLPI state. EL2
cannot impersonate the guest an read ICV_IAR1_EL1 (because it
conveniently has the same encoding as ICC_IAR1_EL1), and if it could,
it would be *destructive* (not what you want). The equivalent of the
LR that is used to hold the highest priority VLPI presented to the
virtual CPU interface is not visible to SW at all.

There are exactly two ways for the hypervisor to get a hint about the
VLPI state (and that's only a hint, as everything can be spurious):

- Make the vPE non resident and use GICR_VPENDBASER.PendingLast bit to
  find out whether there are pending VLPIs

- Make the vPE non resident and get a doorbell interrupt

See the common pattern?

There is no polling mechanism, and the only way to flush the VLPI
state to memory is to destroy the GIC view of the vPE, which is a bit
counter-productive. It also only work on GICv4.1, and not GICv4 (which
is why we don't support live migration on GICv4).

> Sorry for the many stupid questions I'm asking lately, but I'm trying
> to pay more attention to ARM and understand the VGIC and EL1/EL2 split
> better.

Feel free to ask any question. The more people understand how the
architecture works, the better.

	M.

-- 
Without deviation from the norm, progress is not possible.