Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <a5b07aa9-96ea-a9b5-13db-e5dcbd7760e6@intel.com>
 <BEA3CCB8-5127-4E6A-9696-E293C00BFA82@amacapital.net> <CABV8kRxqcCmPKtX3DDOf+47Re1hO1gMeUPhCd6HtDP0-SpcSBw@mail.gmail.com>
 <CALCETrVmsWZ+w6C4RV50DWoL0Qaiy+S6BtXr=QKQEg3MYgAc6w@mail.gmail.com>
In-Reply-To: <CALCETrVmsWZ+w6C4RV50DWoL0Qaiy+S6BtXr=QKQEg3MYgAc6w@mail.gmail.com>
From:   Keno Fischer <keno@juliacomputing.com>
Date:   Tue, 14 Apr 2020 20:09:53 -0400
Message-ID: <CABV8kRxfMNxzy8r2P4GKj-22i+GMj=VDfPWAZB-VSgfrsQsjCA@mail.gmail.com>
Subject: Re: [RFC PATCH v2] x86/arch_prctl: Add ARCH_SET_XCR0 to set XCR0 per-thread
To:     Andy Lutomirski <luto@kernel.org>
Cc:     Dave Hansen <dave.hansen@intel.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>,
        "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" <x86@kernel.org>,
        "H. Peter Anvin" <hpa@zytor.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Andi Kleen <andi@firstfloor.org>,
        Kyle Huey <khuey@kylehuey.com>,
        "Robert O'Callahan" <robert@ocallahan.org>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

> (why would you want to record with an unusual XCR0?)

I was indeed primarily thinking about the replay case when I
originally wrote this patch, but the reason for that is that the machine
that I tend to replay on is fairly new, so the traces we get tend
to be compatible. I have encountered the opposite situation also
where I wanted to send somebody a trace for them to debug, but
their machine didn't support AVX512, so I wanted to disable it.
A while ago we also encountered a similar situation where
PKRU became available on AWS so we couldn't
replay traces from there anymore.
We do of course have CPUID faulting here (which well-behaved
userspace software tends to respect), but the differing return
value from xgetbv messed up the recording nonetheless. In
order to actually make that work, masking those bits out from
XCR0 would have been required. I'm also thinking about the
future where there may be more diversity in XCR0 user state
components and what chips support what.

I assume the objection here will be that we can have our users
reboot with a different kernel command line, which is true enough
and what I will recommend to people for the time being. That said,
part of the appeal for users of rr is that it doesn't require privilege,
so it works in locked down environments like HPC clusters where
users can't easily change the kernel parameters (even KVM can
be an ask here - but is in my experience easier to negotiate with
the sysadmins).

> and replay would use KVM.

Yes, there is still a bit of state management left (e.g. we currently
rely on the kernel for mmap, fork/clone semantics and signal
delivery for deterministic signals) - though of course emulating
all that is much easier than the record side.

> I'm not sure about diversions.

Diversions (terminology note for those not familiar with record
and replay systems - a diversion is a replay that gets turned
back into a real process for the purpose of investigating the
state - think invoking a printing function from the debugger)
are similar to record in that they do handle syscalls (e.g.
for open/read/write for printing, mmap if the process allocates
during a diversion, etc.). We do have more performance
leeway for diversions than we do for records, so we might
be okay without getting too deep into KVM hacking.

I think my overall take-away is that KVM is probably feasible
for replay and diversions, but would require large patches
to KVM to be feasible for record. Unfortunately, that's
also the opposite of where getting a patch like this
into the mainline kernel would be most useful for us.
We do tend to have control over the replay machines,
so if we need to carry a patch like this, it's feasible (as
indeed we have been for the past two years), though
of course we'd like to enable this also for people who
don't want to build their own kernel (as well as for people
for whom that assumption doesn't hold - the assertion that
we control the replay machine is probably quite selfish in
how we in particular use this technology).

Hope that helps,
Keno