On Mon, 2018-01-29 at 13:45:07 -0800, Eduardo Habkost wrote:
> Maybe a generic "family/model/stepping/microcode really matches
> the CPU you are running on" bit would be useful. The bit could
> be enabled only on host-passthrough (aka "-cpu host") mode.
>
> If we really want to be able to migrate to host with different
> CPU models (except Skylake), we could add a more specific "we
> promise the host CPU is never going to be Skylake" bit.
>
> Now, if the hypervisor is not providing any of those bits, I
> would advise against trusting family/model/stepping/microcode
> under a hypervisor. Using a pre-defined CPU model (that doesn't
> necessarily match the host) is very common when using KVM VM
> management stacks.
>
Eduardo,
I don't see how this is possible in a modern virtualization environment.
Under VMware, a VM will be migrated to SkyLake if one is in the cluster and supports the features exposed to the VM. This can occur for suspend/resume as well.
The migration pool isn't a constant. Hosts can be added to a cluster and VMs can be instructed to move across clusters. So there doesn't need to be a SkyLake around when the VM powers on in order for it to eventually end up on a SkyLake.
Even if we expose bit to indicate that FMS matches the underlying host, when does the guest know to query that? The VM can be moved at any point in time, including after the guest asks if FMS matches host.
My apologies for posting onto the mailing list out of the blue. Someone asked my opinion on this suggestion. I'm definitely interested in figuring out whether Linux can fully mitigate the SkyLake RSB problem in virtual environments, but it's not clear how best to achieve that.
Thanks,
David Dunn
> Even if we expose bit to indicate that FMS matches the underlying host, when does the guest know to query that? The VM can be moved at any point in time, including after the guest asks if FMS matches host.
There's no way to enable these mitigations later, so if you always
have to enable the super set of all the mitigations for all the hosts you
might be migrating too.
As of currently that means if you want to ever migrate to Skylake you should
set the Skylake model number and you're good.
-Andi
And if we expect to introduce Cascade Lake into the pool in the
future, we use a Cascade Lake model number?
It sounds like you are suggesting that we set the model number to the
highest model number that will ever be introduced into the pool, at
any time in the future. That approach would also fail the
'is_skylake_era()' test. (Not to mention that we have no idea what
Intel's highest compatible model number will be.)
On Mon, Jan 29, 2018 at 2:41 PM, Andi Kleen <[email protected]> wrote:
>> Even if we expose bit to indicate that FMS matches the underlying host, when does the guest know to query that? The VM can be moved at any point in time, including after the guest asks if FMS matches host.
>
> There's no way to enable these mitigations later, so if you always
> have to enable the super set of all the mitigations for all the hosts you
> might be migrating too.
>
> As of currently that means if you want to ever migrate to Skylake you should
> set the Skylake model number and you're good.
>
> -Andi
(Apologies as I was brought into this thread late, but I believe I have
context).
Could a new "feature" be enumerated on Skylake and beyond which specifies that
a particular problem exists which requires different mitigation than on
previous processors? Perhaps a CPUID bit enumerating this feature (along side
IBRS, IBPB and STIBP) could be exposed on only the newer CPUs. System software
could then query this to know what form of mitigation is necessary.
This could be over-reported in virtualized environments (e.g. a Nehalem CPU
could be represented as needing the Skylake mitigation), such that sometimes
the heavier Skylake+ mitigation would be applied on older CPUs. This is
correct, just slower.
I'm just suggesting this rather than keying on Family/Model/Stepping to avoid
breaking virtual machine migration, et cetera.
Thanks,
Fred, sticking his neck out.
On Jan 29 2:29PM, David Dunn wrote:
> On Mon, 2018-01-29 at 13:45:07 -0800, Eduardo Habkost wrote:
>
> > Maybe a generic "family/model/stepping/microcode really matches
> > the CPU you are running on" bit would be useful. The bit could
> > be enabled only on host-passthrough (aka "-cpu host") mode.
> >
> > If we really want to be able to migrate to host with different
> > CPU models (except Skylake), we could add a more specific "we
> > promise the host CPU is never going to be Skylake" bit.
> >
> > Now, if the hypervisor is not providing any of those bits, I
> > would advise against trusting family/model/stepping/microcode
> > under a hypervisor. Using a pre-defined CPU model (that doesn't
> > necessarily match the host) is very common when using KVM VM
> > management stacks.
> >
>
> Eduardo,
>
> I don't see how this is possible in a modern virtualization environment.
>
> Under VMware, a VM will be migrated to SkyLake if one is in the cluster and supports the features exposed to the VM. This can occur for suspend/resume as well.
>
> The migration pool isn't a constant. Hosts can be added to a cluster and VMs can be instructed to move across clusters. So there doesn't need to be a SkyLake around when the VM powers on in order for it to eventually end up on a SkyLake.
>
> Even if we expose bit to indicate that FMS matches the underlying host, when does the guest know to query that? The VM can be moved at any point in time, including after the guest asks if FMS matches host.
>
> My apologies for posting onto the mailing list out of the blue. Someone asked my opinion on this suggestion. I'm definitely interested in figuring out whether Linux can fully mitigate the SkyLake RSB problem in virtual environments, but it's not clear how best to achieve that.
>
> Thanks,
>
> David Dunn
>
On Mon, Jan 29, 2018 at 10:29:28PM +0000, David Dunn wrote:
> On Mon, 2018-01-29 at 13:45:07 -0800, Eduardo Habkost wrote:
>
> > Maybe a generic "family/model/stepping/microcode really matches
> > the CPU you are running on" bit would be useful. The bit could
> > be enabled only on host-passthrough (aka "-cpu host") mode.
> >
> > If we really want to be able to migrate to host with different
> > CPU models (except Skylake), we could add a more specific "we
> > promise the host CPU is never going to be Skylake" bit.
> >
> > Now, if the hypervisor is not providing any of those bits, I
> > would advise against trusting family/model/stepping/microcode
> > under a hypervisor. Using a pre-defined CPU model (that doesn't
> > necessarily match the host) is very common when using KVM VM
> > management stacks.
> >
>
> Eduardo,
>
> I don't see how this is possible in a modern virtualization
> environment.
>
> Under VMware, a VM will be migrated to SkyLake if one is in the
> cluster and supports the features exposed to the VM. This can
> occur for suspend/resume as well.
>
> The migration pool isn't a constant. Hosts can be added to a
> cluster and VMs can be instructed to move across clusters. So
> there doesn't need to be a SkyLake around when the VM powers on
> in order for it to eventually end up on a SkyLake.
If this is the case for your deployment, this means the guest
must never assume it won't run on a Skylake host (even if f/m/s
is not Skylake), doesn't it? Then the hypervisor won't set the
"we promise the host CPU is never going to be Skylake" bit.
>
> Even if we expose bit to indicate that FMS matches the
> underlying host, when does the guest know to query that? The
> VM can be moved at any point in time, including after the guest
> asks if FMS matches host.
If the VM can be moved at any point of time to a different model
of host CPU, this means you won't tell the guest it can trust
f/m/s because it doesn't represent the underlying host. You
won't set the "f/m/s/m really matches the host CPU" bit.
On both scenarios you describe above, it sounds like Linux must
assume it could migrated to a Skylake host at any moment. This
is exactly why I'm proposing those extra bits.
--
Eduardo
On Mon, Jan 29, 2018 at 02:49:51PM -0800, Jim Mattson wrote:
> And if we expect to introduce Cascade Lake into the pool in the
> future, we use a Cascade Lake model number?
>
> It sounds like you are suggesting that we set the model number to the
> highest model number that will ever be introduced into the pool, at
> any time in the future. That approach would also fail the
> 'is_skylake_era()' test. (Not to mention that we have no idea what
> Intel's highest compatible model number will be.)
Exactly, that's why virtualization and live-migration break the
model of just checking f/m/s/microcode: the guest doesn't need to
work around bugs that are present in the current host, but the
set of bugs that could appear on any future host it can run on.
>
> On Mon, Jan 29, 2018 at 2:41 PM, Andi Kleen <[email protected]> wrote:
> >> Even if we expose bit to indicate that FMS matches the underlying host, when does the guest know to query that? The VM can be moved at any point in time, including after the guest asks if FMS matches host.
> >
> > There's no way to enable these mitigations later, so if you always
> > have to enable the super set of all the mitigations for all the hosts you
> > might be migrating too.
> >
> > As of currently that means if you want to ever migrate to Skylake you should
> > set the Skylake model number and you're good.
> >
> > -Andi
--
Eduardo
Eduardo,
This is why it would be good to have a CPUID bit that says: "apply SkyLake RSB stuffing." That's preferable to "trust FMS" for VMware.
If Intel defines such a feature flag, sets it on SkyLake, and Linux uses it... that would be very helpful for VMware.
I won't speak for GCE and AWS. But hopefully they can indicate whether it would help them as well.
If Intel cannot define/implement such a flag on SkyLake, then maybe the engineers on this email could define a flag in the hypervisor specific CPUID space. Linux would need to query that flag if it sees CPUID[1].ECX[31] set. That's not as nice since it makes detection on bare metal and virtualization platforms different, but it better than keying off FMS.
David Dunn
On 1/29/18, 5:11 PM, "Eduardo Habkost" <[email protected]> wrote:
On Mon, Jan 29, 2018 at 02:49:51PM -0800, Jim Mattson wrote:
> And if we expect to introduce Cascade Lake into the pool in the
> future, we use a Cascade Lake model number?
>
> It sounds like you are suggesting that we set the model number to the
> highest model number that will ever be introduced into the pool, at
> any time in the future. That approach would also fail the
> 'is_skylake_era()' test. (Not to mention that we have no idea what
> Intel's highest compatible model number will be.)
Exactly, that's why virtualization and live-migration break the
model of just checking f/m/s/microcode: the guest doesn't need to
work around bugs that are present in the current host, but the
set of bugs that could appear on any future host it can run on.
>
> On Mon, Jan 29, 2018 at 2:41 PM, Andi Kleen <[email protected]> wrote:
> >> Even if we expose bit to indicate that FMS matches the underlying host, when does the guest know to query that? The VM can be moved at any point in time, including after the guest asks if FMS matches host.
> >
> > There's no way to enable these mitigations later, so if you always
> > have to enable the super set of all the mitigations for all the hosts you
> > might be migrating too.
> >
> > As of currently that means if you want to ever migrate to Skylake you should
> > set the Skylake model number and you're good.
> >
> > -Andi
--
Eduardo
On Tue, Jan 30, 2018 at 01:20:52AM +0000, David Dunn wrote:
> Eduardo,
>
> This is why it would be good to have a CPUID bit that says:
> "apply SkyLake RSB stuffing." That's preferable to "trust FMS"
> for VMware.
Agreed it would be more useful than "trust FMS". However, I
believe a "no need to apply Skylake RSB stuffing" bit (which I
called "we promise we won't migrate to Skylake" previously) would
allow guests to enable safer behavior by default under older
hypervisors that don't support this bit.
>
> If Intel defines such a feature flag, sets it on SkyLake, and
> Linux uses it... that would be very helpful for VMware.
>
> I won't speak for GCE and AWS. But hopefully they can indicate
> whether it would help them as well.
I agree that having a standard flag on the CPUID space to specify
that would be very helpful.
>
> If Intel cannot define/implement such a flag on SkyLake, then
> maybe the engineers on this email could define a flag in the
> hypervisor specific CPUID space. Linux would need to query
> that flag if it sees CPUID[1].ECX[31] set. That's not as nice
> since it makes detection on bare metal and virtualization
> platforms different, but it better than keying off FMS.
Agreed.
--
Eduardo