On Wednesday, March 16, 2005 7:20 PM Benjamin Herrenschmidt wrote:
>> What mechanism (message??) is used to perform the bus and/or link
>> level reset? For PCI Express the reset is performed by the upstream
>> port driver. My API takes this into account. Are you assuming the
PCI
>> device on the bus does the reset or will there be a PCI bus driver
that
>> will do the reset? How will the PCI error handling code initiate a
>> reset?
>
>The "caller", that is the error management framework. I'm defining the
>API at the driver level, not the implementation at the core level.
>
>For example, on IBM pSeries with PCI-Express, we will probably not have
>an AER driver. This will be all dealt by the firmware which will mimmic
>that to the existing EEH error management. We'll have the same API to
do
>the reset that we have today for resetting a slot.
We decided to implement PCI Express error handling based on the PCI
Express specification in a platform independent manner. This allows any
platform that implements PCI Express AER per the PCI SIG specification
can take advantage of the advanced features, much like SHPC hot-plug or
PCI Express hot-plug implementations.
>You may have noticed in general that I didn't either define who is
>callign those callbacks. It's all implicit that this is done by
platform
>error management code. For example, on ppc64, even the recovery step
>requires action from the platform since the slot has been physically
>isolated. After we have notified all drivers with the "error detected"
>callback, if we decide we can try the "recover" step (all drivers
>returned they could try it and we decided the error wasn't too fatal)
we
>will call the firmware to re-enable IOs on the slot and call the
>"recover" step.
For PCI Express the endpoint device driver can take recovery action on
its own, depending on the nature of the error so long as it does not
affect the upstream device. This can include endpoint device resets.
We expect the driver to do this upon error notification, if possible.
In PCI Express since the driver will have the most knowledge regarding
the error it will have the best ability to do device dependent recovery
and IO retry. If its recovery fails then the AER driver will ask the
upstream device driver to perform the link reset. Since this is more of
a side effect an explicit call to recover is not necessary. However, we
understand and agree that it is needed to support the general error
recovery cases for PCI.
To support the AER driver calling an upstream device to initiate a reset
of the link we need a specific callback since the driver doing the reset
is not the driver who got the error. In the case of general PCI this
could be useful if a PCI bus driver were available to support the
callback for a bridge device. This would also support specific error
recovery calls to reset an endpoint adapter. We need a call to request
a driver to perform a reset on a link or device.
Thanks,
Long
On Thu, 2005-03-17 at 10:53 -0800, Nguyen, Tom L wrote:
> To support the AER driver calling an upstream device to initiate a reset
> of the link we need a specific callback since the driver doing the reset
> is not the driver who got the error. In the case of general PCI this
> could be useful if a PCI bus driver were available to support the
> callback for a bridge device. This would also support specific error
> recovery calls to reset an endpoint adapter. We need a call to request
> a driver to perform a reset on a link or device.
That is quite implementation specific, it doesn't need to be part of the
API (the way the general error management is implemented in PCIE could
be completely done within the bus drivers I suppose). Again, I'm not
trying to define or force a given implementation. I'm trying to define
the driver-side API, that's all.
I have difficulties following all of your previous explanations, I must
admit. My point here is I'd like you to find out if the API can fit on
the driver side, and if not, what would need to be changed. For example,
we might want to distinguish between slot reset (full hard reset) and
link reset, that sort of thing (thus adding a new state for link reset
and a new return code for the others for requesting a link reset if
possible, platforms that don't do it, like IBM EEH PCI would just
fallback to full reset).
Again, the goal here is to have a way for drivers to be mostly bus
agnostic (that is not have to care if they are running on PCI, PCI-X,
PCIE, with or without IBM EEH mecanism, and whatever other mecanism
another vendor might provide) and still implement basic error recovery.
A driver _designed_ for a PCI-Express deviec that knows it's on PCI
Express can perfectly use additional APIs to gather more error details,
etc... but it would be nice to fit the "common needs" as much as
possible in a common and _SIMPLE_ API. The simplicity here is a
requirement, I'm very serious about it, because if it's not simple,
drivers either won't implement it or won't get it right.
Ben.
Nguyen, Tom L writes:
> We decided to implement PCI Express error handling based on the PCI
> Express specification in a platform independent manner. This allows any
> platform that implements PCI Express AER per the PCI SIG specification
> can take advantage of the advanced features, much like SHPC hot-plug or
> PCI Express hot-plug implementations.
Does the PCI Express AER specification define an API for drivers?
> For PCI Express the endpoint device driver can take recovery action on
> its own, depending on the nature of the error so long as it does not
> affect the upstream device. This can include endpoint device resets.
Likewise, with EEH the device driver could take recovery action on its
own. But we don't want to end up with multiple sets of recovery code
in drivers, if possible. Also we want the recovery code to be as
simple as possible, otherwise driver authors will get it wrong.
> To support the AER driver calling an upstream device to initiate a reset
> of the link we need a specific callback since the driver doing the reset
> is not the driver who got the error. In the case of general PCI this
I would see the AER driver as being included in the "platform" code.
The AER driver would be be closely involved in the recovery process.
What is the state of a link during the time between when an error is
detected and when a link reset is done? Is the link usable? What
happens if you try to do a MMIO read from a device downstream of the
link?
Regards,
Paul.