Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <a5b07aa9-96ea-a9b5-13db-e5dcbd7760e6@intel.com> <BEA3CCB8-5127-4E6A-9696-E293C00BFA82@amacapital.net>
In-Reply-To: <BEA3CCB8-5127-4E6A-9696-E293C00BFA82@amacapital.net>
From:   Keno Fischer <keno@juliacomputing.com>
Date:   Tue, 7 Apr 2020 14:30:13 -0400
Message-ID: <CABV8kRxqcCmPKtX3DDOf+47Re1hO1gMeUPhCd6HtDP0-SpcSBw@mail.gmail.com>
Subject: Re: [RFC PATCH v2] x86/arch_prctl: Add ARCH_SET_XCR0 to set XCR0 per-thread
To:     Andy Lutomirski <luto@amacapital.net>
Cc:     Dave Hansen <dave.hansen@intel.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>,
        "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" <x86@kernel.org>,
        "H. Peter Anvin" <hpa@zytor.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Andi Kleen <andi@firstfloor.org>,
        Kyle Huey <khuey@kylehuey.com>,
        "Robert O'Callahan" <robert@ocallahan.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

> TSX!

Yes, it's problematic, but luckily turns out to
be ok in practice if masked off in cpuid.

> I think rr should give the raw KVM API at least a try.  It should be poss=
ible to fire up a vCPU in CPL3 in the correct state.  No guest kernel requi=
red.  I don=E2=80=99t know if there will be issues with the perf API, thoug=
h.

Yes, I've looked into it, but stopped short of doing a
complete implementation. Using KVM to solve it
for replay would probably be feasible with a moderate
amount of engineering work, since rr does very few
syscalls during replay. I'm a bit afraid of the
performance implications, but I don't have numbers on this.

Record and diversions are a lot harder though, because
in this mode the tracee is a live process and able to do
syscalls (and needs to receive signals and all that good
stuff associated with being a real process). For diversions,
performance isn't super important, so we could probably
emulate this, but for record, performance is quite critical.
I assume it would be possible to add a feature to KVM
where it forwards syscalls made in guest CPL3 to the real
kernel without round-trip through userspace, but I'm just
seeing myself back here asking
for a weird KVM feature that nobody but me wants ;)
(well almost nobody, as I mentioned, there's an
academic project that tried this with a custom kernel
plugin - http://dune.scs.stanford.edu/).

Admittedly, the use case for this feature during record is
less pressing, since in our (operational) case
the replay machines tend to be much newer than
the record machines, but I wouldn't be surprised if I got
bit by this as soon as the next user xstate component gets
added and users start sending me those kinds of traces,
even if we mask off the feature in CPUID (which rr already
supports for record for similar reasons).