Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <a5b07aa9-96ea-a9b5-13db-e5dcbd7760e6@intel.com>
 <BEA3CCB8-5127-4E6A-9696-E293C00BFA82@amacapital.net> <CABV8kRxqcCmPKtX3DDOf+47Re1hO1gMeUPhCd6HtDP0-SpcSBw@mail.gmail.com>
In-Reply-To: <CABV8kRxqcCmPKtX3DDOf+47Re1hO1gMeUPhCd6HtDP0-SpcSBw@mail.gmail.com>
From:   Andy Lutomirski <luto@kernel.org>
Date:   Tue, 14 Apr 2020 16:20:14 -0700
Message-ID: <CALCETrVmsWZ+w6C4RV50DWoL0Qaiy+S6BtXr=QKQEg3MYgAc6w@mail.gmail.com>
Subject: Re: [RFC PATCH v2] x86/arch_prctl: Add ARCH_SET_XCR0 to set XCR0 per-thread
To:     Keno Fischer <keno@juliacomputing.com>
Cc:     Dave Hansen <dave.hansen@intel.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>,
        "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" <x86@kernel.org>,
        "H. Peter Anvin" <hpa@zytor.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Andi Kleen <andi@firstfloor.org>,
        Kyle Huey <khuey@kylehuey.com>,
        "Robert O'Callahan" <robert@ocallahan.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Apr 7, 2020 at 11:30 AM Keno Fischer <keno@juliacomputing.com> wrot=
e:
>
> > TSX!
>
> Yes, it's problematic, but luckily turns out to
> be ok in practice if masked off in cpuid.
>
> > I think rr should give the raw KVM API at least a try.  It should be po=
ssible to fire up a vCPU in CPL3 in the correct state.  No guest kernel req=
uired.  I don=E2=80=99t know if there will be issues with the perf API, tho=
ugh.
>
> Yes, I've looked into it, but stopped short of doing a
> complete implementation. Using KVM to solve it
> for replay would probably be feasible with a moderate
> amount of engineering work, since rr does very few
> syscalls during replay. I'm a bit afraid of the
> performance implications, but I don't have numbers on this.
>
> Record and diversions are a lot harder though, because
> in this mode the tracee is a live process and able to do
> syscalls (and needs to receive signals and all that good
> stuff associated with being a real process). For diversions,
> performance isn't super important, so we could probably
> emulate this, but for record, performance is quite critical.
> I assume it would be possible to add a feature to KVM
> where it forwards syscalls made in guest CPL3 to the real
> kernel without round-trip through userspace, but I'm just
> seeing myself back here asking
> for a weird KVM feature that nobody but me wants ;)
> (well almost nobody, as I mentioned, there's an
> academic project that tried this with a custom kernel
> plugin - http://dune.scs.stanford.edu/).
>
> Admittedly, the use case for this feature during record is
> less pressing, since in our (operational) case
> the replay machines tend to be much newer than
> the record machines, but I wouldn't be surprised if I got
> bit by this as soon as the next user xstate component gets
> added and users start sending me those kinds of traces,
> even if we mask off the feature in CPUID (which rr already
> supports for record for similar reasons).

I'm imagining that rr would do record the usual way with normal XCR0
(why would you want to record with an unusual XCR0?) and replay would
use KVM.  I'm not sure about diversions.  This way KVM wouldn't need
to deal with syscalls.

Would this work?