Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Sat, 22 Aug 2020 11:31:54 +0100
Message-ID: <87h7svm0o5.wl-maz@kernel.org>
From:   Marc Zyngier <maz@kernel.org>
To:     Steven Price <steven.price@arm.com>
Cc:     Keqian Zhu <zhukeqian1@huawei.com>, linux-kernel@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu,
        kvm@vger.kernel.org, Catalin Marinas <catalin.marinas@arm.com>,
        Will Deacon <will@kernel.org>,
        James Morse <james.morse@arm.com>,
        Suzuki K Poulose <suzuki.poulose@arm.com>,
        wanghaibin.wang@huawei.com
Subject: Re: [RFC PATCH 0/5] KVM: arm64: Add pvtime LPT support
In-Reply-To: <f14cfd5b-c103-5d56-82fb-59d0371c6f21@arm.com>
References: <20200817084110.2672-1-zhukeqian1@huawei.com>
        <8308f52e4c906cad710575724f9e3855@kernel.org>
        <f14cfd5b-c103-5d56-82fb-59d0371c6f21@arm.com>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL/10.8 EasyPG/1.0.0 Emacs/26.3
 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Hi Steven,

On Wed, 19 Aug 2020 09:54:40 +0100,
Steven Price <steven.price@arm.com> wrote:
>=20
> On 18/08/2020 15:41, Marc Zyngier wrote:
> > On 2020-08-17 09:41, Keqian Zhu wrote:
> >> Hi all,
> >>=20
> >> This patch series picks up the LPT pvtime feature originally developed
> >> by Steven Price: https://patchwork.kernel.org/cover/10726499/
> >>=20
> >> Backgroud:
> >>=20
> >> There is demand for cross-platform migration, which means we have to
> >> solve different CPU features and arch counter frequency between hosts.
> >> This patch series can solve the latter problem.
> >>=20
> >> About LPT:
> >>=20
> >> This implements support for Live Physical Time (LPT) which provides the
> >> guest with a method to derive a stable counter of time during which the
> >> guest is executing even when the guest is being migrated between hosts
> >> with different physical counter frequencies.
> >>=20
> >> Changes on Steven Price's work:
> >> 1. LPT structure: use symmatical semantics of scale multiplier, and use
> >> =C2=A0=C2=A0 fraction bits instead of "shift" to make everything clear.
> >> 2. Structure allocation: host kernel does not allocates the LPT
> >> structure,
> >> =C2=A0=C2=A0 instead it is allocated by userspace through VM attribute=
s. The
> >> save/restore
> >> =C2=A0=C2=A0 functionality can be removed.
> >> 3. Since LPT structure just need update once for each guest run,
> >> add a flag to
> >> =C2=A0=C2=A0 indicate the update status. This has two benifits: 1) avo=
id
> >> multiple update
> >> =C2=A0=C2=A0 by each vCPUs. 2) If the update flag is not set, then ret=
urn NOT
> >> SUPPORT for
> >> =C2=A0=C2=A0 coressponding guest HVC call.
> >> 4. Add VM device attributes interface for userspace configuration.
> >> 5. Add a base LPT read/write layer to reduce code.
> >> 6. Support ptimer scaling.
> >> 7. Support timer event stream translation.
> >>=20
> >> Things need concern:
> >> 1. https://developer.arm.com/docs/den0057/a needs update.
> >=20
> > LPT was explicitly removed from the spec because it doesn't really
> > solve the problem, specially for the firmware: EFI knows
> > nothing about this, for example. How is it going to work?
> > Also, nobody was ever able to explain how this would work for
> > nested virt.
> >=20
> > ARMv8.4 and ARMv8.6 have the feature set that is required to solve
> > this problem without adding more PV to the kernel.
>=20
> Hi Marc,
>=20
> These are good points, however we do still have the situation that
> CPUs that don't have ARMv8.4/8.6 clearly cannot implement this. I
> presume the use-case Keqian is looking at predates the necessary
> support in the CPU - Keqian if you can provide more details on the
> architecture(s) involved that would be helpful.

My take on this is that it is a fictional use case. In my experience,
migration happens across *identical* systems, and *any* difference
visible to guests will cause things to go wrong. Errata management
gets in the way, as usual (name *one* integration that isn't broken
one way or another!).

Allowing migration across heterogeneous hosts requires a solution to
the errata management problem, which everyone (including me) has
decided to ignore so far (and I claim that not having a constant timer
frequency exposed to guests is an architecture bug).

> Nested virt is indeed more of an issue - we did have some ideas around
> using SDEI that never made it to the spec.

SDEI? Sigh... Why would SDEI be useful for NV and not for !NV?

> However I would argue that the most pragmatic approach would be to
> not support the combination of nested virt and LPT. Hopefully that
> can wait until the counter scaling support is available and not
> require PV.

And have yet another set of band aids that paper over the fact that we
can't get a consistent story on virtualization? No, thank you.

NV is (IMHO) much more important than LPT as it has a chance of
getting used. LPT is just another tick box, and the fact that ARM is
ready to ignore sideline a decent portion of the architecture is a
clear sign that it hasn't been thought out.

> We are discussing (re-)releasing the spec with the LPT parts added. If
> you have fundamental objections then please me know.

I do, see above. I'm stating that the use case doesn't really exist
given the state of the available HW and the fragmentation of the
architecture, and that ignoring the most important innovation in the
virtualization architecture since ARMv7 is at best short-sighted.

Time scaling is just an instance of the errata management problem, and
that is the issue that needs solving. Papering over part of the
problem is not helping.

	M.

--=20
Without deviation from the norm, progress is not possible.