Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From:   Andy Lutomirski <luto@amacapital.net>
Mime-Version: 1.0 (1.0)
Subject: Re: [RFC PATCH v2] x86/arch_prctl: Add ARCH_SET_XCR0 to set XCR0 per-thread
Date:   Tue, 7 Apr 2020 07:16:34 -0700
Message-Id: <BEA3CCB8-5127-4E6A-9696-E293C00BFA82@amacapital.net>
References: <a5b07aa9-96ea-a9b5-13db-e5dcbd7760e6@intel.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Keno Fischer <keno@juliacomputing.com>,
        linux-kernel@vger.kernel.org, Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, x86@kernel.org,
        "H. Peter Anvin" <hpa@zytor.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Andi Kleen <andi@firstfloor.org>,
        Kyle Huey <khuey@kylehuey.com>,
        Robert O'Callahan <robert@ocallahan.org>
In-Reply-To: <a5b07aa9-96ea-a9b5-13db-e5dcbd7760e6@intel.com>
To:     Dave Hansen <dave.hansen@intel.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


> On Apr 7, 2020, at 7:07 AM, Dave Hansen <dave.hansen@intel.com> wrote:
>=20
> =EF=BB=BFOn 4/7/20 5:21 AM, Peter Zijlstra wrote:
>> You had a fairly long changelog detailing what the patchd does; but I've
>> failed to find a single word on _WHY_ we want to do any of that.
>=20
> The goal in these record/replay systems is to be able to recreate thee
> exact same program state on two systems at two different times.  To make
> it reasonably fast, they try to minimize the number of snapshots they
> have to take and avoid things like single stepping.
>=20
> So, there are some windows where they just let the CPU run and don't
> bother with taking any snapshots of register state, for instance.  Let's
> say you read a word from shared memory, multiply it and shift it around
> some registers, then stick it back in shared memory.  Most of these
> things will just a record the snapshot at the memory read and assume
> that all the instructions in the middle execute deterministically.  That
> eliminates a ton of snapshots.
>=20
> But, what if an instruction in the middle isn't deterministic between
> two machines.  Let's say you record a trace on a a Broadwell system,
> then try to replay it on a Skylake, and one of the non-snapshotted
> instructions is xgetbv.  Skylake added MPX, so xgetbv will return
> different values.  Your replay diverges from what was "recorded", and
> life sucks.
>=20
> Same problem exists for CPUID, but that was hacked around in another set.
>=20
> I'm also trying to think of what kinds of things CPU companies add to
> their architectures that would break this stuff.  I can't recall ever
> having a discussion with folks at Intel where we're designing a CPU
> feature and we say, "Can't do that, it would break record/replay".  I
> suspect there are more of these landmines around and I bet that we're
> building more of them into CPUs every day.

TSX!

I think rr should give the raw KVM API at least a try.  It should be possibl=
e to fire up a vCPU in CPL3 in the correct state.  No guest kernel required.=
  I don=E2=80=99t know if there will be issues with the perf API, though.

If we actually do merge this XCR0 hack, I think the rule should be that it h=
as no effect on kernel behavior.  Signals, ptrace, etc reflect the normal XC=
R0, not the overridden value.=