Add support for error containment when a PCI pass-thru device assigned to a KVM
guest encounters an error. This is for PCIe devices/drivers that support AER
functionality. When the OS is notified of an error in a device either
through the firmware first approach or through an interrupt handled by the AER
root port driver, concerned subsystems are notified by invoking callbacks
registered by these subsystems. The device is also marked as tainted till the
corresponding driver recovery routines are successful.
KVM module registers for a notification of such errors. In the KVM callback
routine, a global counter is incremented to keep track of the error
notification. Before each CPU enters guest mode to execute guest code,
appropriate checks are done to see if the impacted device belongs to the guest
or not. If the device belongs to the guest, qemu hypervisor for the guest is
informed and the guest is immediately brought down, thus preventing or
minimizing chances of any bad data being written out by the guest driver
after the device has encountered an error.
Note that the changes here are specific to PCI pass-thru devices and is
confined to error containment. Error recovery is not included in these set
of changes. A future set of patches is planned to address SR-IOV devices and
VFIO devices assigned to guests as well as recovery without bringing down
the guest.
---
Vijay Mohan Pandarathil(4):
AER-PCI: Add infrastructure for notification of errors to other subsystems
AER-GHES: Add support for error notification in firmware first approach of AER
AER-KVM: Integration of KVM with AER for PCI pass-thru devices
AER-QEMU: Bring down the guest when KVM detects a PCI device error
arch/x86/include/asm/kvm_host.h | 1 +
arch/x86/kvm/x86.c | 44 ++++++++++++++++++++++++++++++++++++++
drivers/acpi/apei/ghes.c | 41 +++++++++++++++++++++++++++++++++++
drivers/pci/pcie/aer/aerdrv.c | 20 +++++++++++++++++
drivers/pci/pcie/aer/aerdrv_core.c | 9 +++++++-
include/linux/aer.h | 4 ++++
include/linux/kvm_host.h | 4 ++++
include/linux/pci.h | 2 ++
include/uapi/linux/kvm.h | 1 +
virt/kvm/assigned-dev.c | 34 +++++++++++++++++++++++++++++
virt/kvm/kvm_main.c | 34 +++++++++++++++++++++++++++++
11 files changed, 193 insertions(+), 1 deletion(-)
Qemu files changed
kvm-all.c | 6 ++++++
linux-headers/linux/kvm.h | 1 +
2 files changed, 7 insertions(+)
On Tue, Nov 20, 2012 at 06:31:48AM +0000, Pandarathil, Vijaymohan R wrote:
> Add support for error containment when a PCI pass-thru device assigned to a KVM
> guest encounters an error. This is for PCIe devices/drivers that support AER
> functionality. When the OS is notified of an error in a device either
> through the firmware first approach or through an interrupt handled by the AER
> root port driver, concerned subsystems are notified by invoking callbacks
> registered by these subsystems. The device is also marked as tainted till the
> corresponding driver recovery routines are successful.
>
> KVM module registers for a notification of such errors. In the KVM callback
> routine, a global counter is incremented to keep track of the error
> notification. Before each CPU enters guest mode to execute guest code,
> appropriate checks are done to see if the impacted device belongs to the guest
> or not. If the device belongs to the guest, qemu hypervisor for the guest is
> informed and the guest is immediately brought down, thus preventing or
> minimizing chances of any bad data being written out by the guest driver
> after the device has encountered an error.
I'm surprised that the hypervisor would shut down the guest when PCIe
AER kicks in for a pass-through device. Shouldn't we pass the AER event
into the guest and deal with it there?
The equivalent to this policy on physical hardware would be that the CPU
is reset or the machine is powered down on AER. That doesn't sound
right.
Stefan
> -----Original Message-----
> From: Stefan Hajnoczi [mailto:[email protected]]
> Sent: Tuesday, November 20, 2012 5:41 AM
> To: Pandarathil, Vijaymohan R
> Cc: [email protected]; [email protected]; [email protected];
> [email protected]
> Subject: Re: [PATCH 0/4] AER-KVM: Error containment of PCI pass-thru
> devices assigned to KVM guests
>
> On Tue, Nov 20, 2012 at 06:31:48AM +0000, Pandarathil, Vijaymohan R wrote:
> > Add support for error containment when a PCI pass-thru device assigned to
> a KVM
> > guest encounters an error. This is for PCIe devices/drivers that support
> AER
> > functionality. When the OS is notified of an error in a device either
> > through the firmware first approach or through an interrupt handled by
> the AER
> > root port driver, concerned subsystems are notified by invoking callbacks
> > registered by these subsystems. The device is also marked as tainted till
> the
> > corresponding driver recovery routines are successful.
> >
> > KVM module registers for a notification of such errors. In the KVM
> callback
> > routine, a global counter is incremented to keep track of the error
> > notification. Before each CPU enters guest mode to execute guest code,
> > appropriate checks are done to see if the impacted device belongs to the
> guest
> > or not. If the device belongs to the guest, qemu hypervisor for the guest
> is
> > informed and the guest is immediately brought down, thus preventing or
> > minimizing chances of any bad data being written out by the guest driver
> > after the device has encountered an error.
>
> I'm surprised that the hypervisor would shut down the guest when PCIe
> AER kicks in for a pass-through device. Shouldn't we pass the AER event
> into the guest and deal with it there?
Agreed. That would be the ideal behavior and is planned in a future patch.
Lack of control over the capabilities/type of the OS/drivers running in
the guest is also a concern in passing along the event to the guest.
My understanding is that in the current implementation of Linux/KVM, these
errors are not handled at all and can potentially cause a guest hang or
crash or even data corruption depending on the implementation of the guest
driver for the device. As a first step, these patches make the behavior
better by doing error containment with a predictable behavior when such
errors occur.
>
> The equivalent to this policy on physical hardware would be that the CPU
> is reset or the machine is powered down on AER. That doesn't sound
> right.
>
> Stefan
On Tue, Nov 20, 2012 at 02:09:46PM +0000, Pandarathil, Vijaymohan R wrote:
>
>
> > -----Original Message-----
> > From: Stefan Hajnoczi [mailto:[email protected]]
> > Sent: Tuesday, November 20, 2012 5:41 AM
> > To: Pandarathil, Vijaymohan R
> > Cc: [email protected]; [email protected]; [email protected];
> > [email protected]
> > Subject: Re: [PATCH 0/4] AER-KVM: Error containment of PCI pass-thru
> > devices assigned to KVM guests
> >
> > On Tue, Nov 20, 2012 at 06:31:48AM +0000, Pandarathil, Vijaymohan R wrote:
> > > Add support for error containment when a PCI pass-thru device assigned to
> > a KVM
> > > guest encounters an error. This is for PCIe devices/drivers that support
> > AER
> > > functionality. When the OS is notified of an error in a device either
> > > through the firmware first approach or through an interrupt handled by
> > the AER
> > > root port driver, concerned subsystems are notified by invoking callbacks
> > > registered by these subsystems. The device is also marked as tainted till
> > the
> > > corresponding driver recovery routines are successful.
> > >
> > > KVM module registers for a notification of such errors. In the KVM
> > callback
> > > routine, a global counter is incremented to keep track of the error
> > > notification. Before each CPU enters guest mode to execute guest code,
> > > appropriate checks are done to see if the impacted device belongs to the
> > guest
> > > or not. If the device belongs to the guest, qemu hypervisor for the guest
> > is
> > > informed and the guest is immediately brought down, thus preventing or
> > > minimizing chances of any bad data being written out by the guest driver
> > > after the device has encountered an error.
> >
> > I'm surprised that the hypervisor would shut down the guest when PCIe
> > AER kicks in for a pass-through device. Shouldn't we pass the AER event
> > into the guest and deal with it there?
>
> Agreed. That would be the ideal behavior and is planned in a future patch.
> Lack of control over the capabilities/type of the OS/drivers running in
> the guest is also a concern in passing along the event to the guest.
>
> My understanding is that in the current implementation of Linux/KVM, these
> errors are not handled at all and can potentially cause a guest hang or
> crash or even data corruption depending on the implementation of the guest
> driver for the device. As a first step, these patches make the behavior
> better by doing error containment with a predictable behavior when such
> errors occur.
For both ACPI notifications and Linux PCI AER driver there is a way for
the PCI driver to receive a notification, correct?
Can just have virt/kvm/assigned-dev.c code register such a notifier (as
a "PCI driver") and then perform appropriate action?
Also the semantics of "tainted driver" is not entirely clear.
Is there any reason for not having this feature for VFIO only, as KVM
device assigment is being phased out?
On Mon, Nov 26, 2012 at 09:46:12PM -0200, Marcelo Tosatti wrote:
> On Tue, Nov 20, 2012 at 02:09:46PM +0000, Pandarathil, Vijaymohan R wrote:
> >
> >
> > > -----Original Message-----
> > > From: Stefan Hajnoczi [mailto:[email protected]]
> > > Sent: Tuesday, November 20, 2012 5:41 AM
> > > To: Pandarathil, Vijaymohan R
> > > Cc: [email protected]; [email protected]; [email protected];
> > > [email protected]
> > > Subject: Re: [PATCH 0/4] AER-KVM: Error containment of PCI pass-thru
> > > devices assigned to KVM guests
> > >
> > > On Tue, Nov 20, 2012 at 06:31:48AM +0000, Pandarathil, Vijaymohan R wrote:
> > > > Add support for error containment when a PCI pass-thru device assigned to
> > > a KVM
> > > > guest encounters an error. This is for PCIe devices/drivers that support
> > > AER
> > > > functionality. When the OS is notified of an error in a device either
> > > > through the firmware first approach or through an interrupt handled by
> > > the AER
> > > > root port driver, concerned subsystems are notified by invoking callbacks
> > > > registered by these subsystems. The device is also marked as tainted till
> > > the
> > > > corresponding driver recovery routines are successful.
> > > >
> > > > KVM module registers for a notification of such errors. In the KVM
> > > callback
> > > > routine, a global counter is incremented to keep track of the error
> > > > notification. Before each CPU enters guest mode to execute guest code,
> > > > appropriate checks are done to see if the impacted device belongs to the
> > > guest
> > > > or not. If the device belongs to the guest, qemu hypervisor for the guest
> > > is
> > > > informed and the guest is immediately brought down, thus preventing or
> > > > minimizing chances of any bad data being written out by the guest driver
> > > > after the device has encountered an error.
> > >
> > > I'm surprised that the hypervisor would shut down the guest when PCIe
> > > AER kicks in for a pass-through device. Shouldn't we pass the AER event
> > > into the guest and deal with it there?
> >
> > Agreed. That would be the ideal behavior and is planned in a future patch.
> > Lack of control over the capabilities/type of the OS/drivers running in
> > the guest is also a concern in passing along the event to the guest.
> >
> > My understanding is that in the current implementation of Linux/KVM, these
> > errors are not handled at all and can potentially cause a guest hang or
> > crash or even data corruption depending on the implementation of the guest
> > driver for the device. As a first step, these patches make the behavior
> > better by doing error containment with a predictable behavior when such
> > errors occur.
>
> For both ACPI notifications and Linux PCI AER driver there is a way for
> the PCI driver to receive a notification, correct?
>
> Can just have virt/kvm/assigned-dev.c code register such a notifier (as
> a "PCI driver") and then perform appropriate action?
>
> Also the semantics of "tainted driver" is not entirely clear.
>
> Is there any reason for not having this feature for VFIO only, as KVM
> device assigment is being phased out?
>
Exactly. We shouldn't add checks to guest entry code and introduce new
userspace ABI to add minor feature to deprecated code. New userspace ABI
means that QEMU changes are needed, so the feature will be fully functional
only with latest QEMU which is capable of using VFIO anyway.
--
Gleb.