This patch adds virtualization support description for DFL based
FPGA devices (based on PCIe SRIOV), and introductions to new
interfaces added by new dfl private feature drivers.
Signed-off-by: Xu Yilun <[email protected]>
Signed-off-by: Wu Hao <[email protected]>
---
v2: update description for thermal/power management user interfaces.
---
Documentation/fpga/dfl.txt | 115 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 115 insertions(+)
diff --git a/Documentation/fpga/dfl.txt b/Documentation/fpga/dfl.txt
index 6df4621..36610e0 100644
--- a/Documentation/fpga/dfl.txt
+++ b/Documentation/fpga/dfl.txt
@@ -84,6 +84,8 @@ The following functions are exposed through ioctls:
Get driver API version (DFL_FPGA_GET_API_VERSION)
Check for extensions (DFL_FPGA_CHECK_EXTENSION)
Program bitstream (DFL_FPGA_FME_PORT_PR)
+ Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN)
+ Release port from PF (DFL_FPGA_FME_PORT_RELEASE)
More functions are exposed through sysfs
(/sys/class/fpga_region/regionX/dfl-fme.n/):
@@ -99,6 +101,24 @@ More functions are exposed through sysfs
one FPGA device may have more than one port, this sysfs interface indicates
how many ports the FPGA device has.
+ Power management (dfl_fme_power hwmon)
+ power management hwmon sysfs interfaces allow user to read power management
+ information (power consumption, thresholds, threshold status, limits, etc.)
+ and and configure power thresholds for different throttling levels.
+
+ Thermal management (dfl_fme_thermal hwmon)
+ thermal management hwmon sysfs interfaces allow user to read thermal
+ management information (current temperature, thresholds, threshold status,
+ etc.).
+
+ Global error reporting management (errors/)
+ error reporting sysfs interfaces allow user to read errors detected by the
+ hardware, and clear the logged errors.
+
+ Performance counters (perf/)
+ performance counters sysfs interfaces allow user to use different counters
+ to get performance data.
+
FIU - PORT
==========
@@ -139,6 +159,10 @@ More functions are exposed through sysfs:
Read Accelerator GUID (afu_id)
afu_id indicates which PR bitstream is programmed to this AFU.
+ Error reporting (errors/)
+ error reporting sysfs interfaces allow user to read port/afu errors
+ detected by the hardware, and clear the logged errors.
+
DFL Framework Overview
======================
@@ -212,6 +236,97 @@ the compat_id exposed by the target FPGA region. This check is usually done by
userspace before calling the reconfiguration IOCTL.
+FPGA virtualization - PCIe SRIOV
+================================
+This section describes the virtualization support on DFL based FPGA device to
+enable accessing an accelerator from applications running in a virtual machine
+(VM). This section only describes the PCIe based FPGA device with SRIOV support.
+
+Features supported by the particular FPGA device are exposed through Device
+Feature Lists, as illustrated below:
+
+ +-------------------------------+ +-------------+
+ | PF | | VF |
+ +-------------------------------+ +-------------+
+ ^ ^ ^ ^
+ | | | |
++-----|------------|---------|--------------|-------+
+| | | | | |
+| +-----+ +-------+ +-------+ +-------+ |
+| | FME | | Port0 | | Port1 | | Port2 | |
+| +-----+ +-------+ +-------+ +-------+ |
+| ^ ^ ^ |
+| | | | |
+| +-------+ +------+ +-------+ |
+| | AFU | | AFU | | AFU | |
+| +-------+ +------+ +-------+ |
+| |
+| DFL based FPGA PCIe Device |
++---------------------------------------------------+
+
+FME is always accessed through the physical function (PF).
+
+Ports (and related AFUs) are accessed via PF by default, but could be exposed
+through virtual function (VF) devices via PCIe SRIOV. Each VF only contains
+1 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators)
+created via PCIe SRIOV interface, to virtual machines.
+
+The driver organization in virtualization case is illustrated below:
+
+ +-------++------++------+ |
+ | FME || FME || FME | |
+ | FPGA || FPGA || FPGA | |
+ |Manager||Bridge||Region| |
+ +-------++------++------+ |
+ +-----------------------+ +--------+ | +--------+
+ | FME | | AFU | | | AFU |
+ | Module | | Module | | | Module |
+ +-----------------------+ +--------+ | +--------+
+ +-----------------------+ | +-----------------------+
+ | FPGA Container Device | | | FPGA Container Device |
+ | (FPGA Base Region) | | | (FPGA Base Region) |
+ +-----------------------+ | +-----------------------+
+ +------------------+ | +------------------+
+ | FPGA PCIE Module | | Virtual | FPGA PCIE Module |
+ +------------------+ Host | Machine +------------------+
+ -------------------------------------- | ------------------------------
+ +---------------+ | +---------------+
+ | PCI PF Device | | | PCI VF Device |
+ +---------------+ | +---------------+
+
+FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device
+is detected. It:
+
+ a) finish enumeration on both FPGA PCIe PF and VF device using common
+ interfaces from DFL framework.
+ b) supports SRIOV.
+
+The FME device driver plays a management role in this driver architecture, it
+provides ioctls to release Port from PF and assign Port to PF. After release
+a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV
+sysfs interface.
+
+To enable accessing an accelerator from applications running in a VM, the
+respective AFU's port needs to be assigned to a VF using the following steps:
+
+ a) The PF owns all AFU ports by default. Any port that needs to be
+ reassigned to a VF must first be released through the
+ DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device.
+
+ b) Once N ports are released from PF, then user can use command below
+ to enable SRIOV and VFs. Each VF owns only one Port with AFU.
+
+ echo N > $PCI_DEVICE_PATH/sriov_numvfs
+
+ c) Pass through the VFs to VMs
+
+ d) The AFU under VF is accessible from applications in VM (using the
+ same driver inside the VF).
+
+Note that an FME can't be assigned to a VF, thus PR and other management
+functions are only available via the PF.
+
+
Device enumeration
==================
This section introduces how applications enumerate the fpga device from
--
1.8.3.1
On Mon, Apr 29, 2019 at 4:12 AM Wu Hao <[email protected]> wrote:
>
> This patch adds virtualization support description for DFL based
> FPGA devices (based on PCIe SRIOV), and introductions to new
> interfaces added by new dfl private feature drivers.
>
> Signed-off-by: Xu Yilun <[email protected]>
> Signed-off-by: Wu Hao <[email protected]>
Acked-by: Alan Tull <[email protected]>
Thanks,
Alan
On Thu, May 16, 2019 at 12:36 PM Alan Tull <[email protected]> wrote:
>
> On Mon, Apr 29, 2019 at 4:12 AM Wu Hao <[email protected]> wrote:
Hi Hao,
Most of this patchset looks ready to go upstream or nearly so with
pretty straightforward changes . Patches 17 and 18 need minor changes
and please change the scnprintf in the other patches. The patches
that had nontrivial changes are the power and thermal ones involving
hwmon. I'm hoping to send up the patchset minus the hwmon patches in
the next version if there's no unforseen issues. If the hwmon patches
are ready then also, that's great, but otherwise those patches don't
need to hold up all the rest of the patchset. How's that sound?
Alan
> >
> > This patch adds virtualization support description for DFL based
> > FPGA devices (based on PCIe SRIOV), and introductions to new
> > interfaces added by new dfl private feature drivers.
> >
> > Signed-off-by: Xu Yilun <[email protected]>
> > Signed-off-by: Wu Hao <[email protected]>
>
> Acked-by: Alan Tull <[email protected]>
>
> Thanks,
> Alan
On Thu, May 16, 2019 at 12:53:00PM -0500, Alan Tull wrote:
> On Thu, May 16, 2019 at 12:36 PM Alan Tull <[email protected]> wrote:
> >
> > On Mon, Apr 29, 2019 at 4:12 AM Wu Hao <[email protected]> wrote:
>
> Hi Hao,
>
> Most of this patchset looks ready to go upstream or nearly so with
> pretty straightforward changes . Patches 17 and 18 need minor changes
> and please change the scnprintf in the other patches. The patches
> that had nontrivial changes are the power and thermal ones involving
> hwmon. I'm hoping to send up the patchset minus the hwmon patches in
> the next version if there's no unforseen issues. If the hwmon patches
> are ready then also, that's great, but otherwise those patches don't
> need to hold up all the rest of the patchset. How's that sound?
Hi Alan
Thanks for your time for reviewing this patchset.
This sounds good to me. Only thing here is, I need to split the patch which
updates documentation into 2 patches (to remove hwmon description in doc),
but for sure, it should be very easy. :)
Thanks
Hao
>
> Alan
>
> > >
> > > This patch adds virtualization support description for DFL based
> > > FPGA devices (based on PCIe SRIOV), and introductions to new
> > > interfaces added by new dfl private feature drivers.
> > >
> > > Signed-off-by: Xu Yilun <[email protected]>
> > > Signed-off-by: Wu Hao <[email protected]>
> >
> > Acked-by: Alan Tull <[email protected]>
> >
> > Thanks,
> > Alan
On Thu, May 16, 2019 at 11:27 PM Wu Hao <[email protected]> wrote:
>
> On Thu, May 16, 2019 at 12:53:00PM -0500, Alan Tull wrote:
> > On Thu, May 16, 2019 at 12:36 PM Alan Tull <[email protected]> wrote:
> > >
> > > On Mon, Apr 29, 2019 at 4:12 AM Wu Hao <[email protected]> wrote:
> >
> > Hi Hao,
> >
> > Most of this patchset looks ready to go upstream or nearly so with
> > pretty straightforward changes . Patches 17 and 18 need minor changes
> > and please change the scnprintf in the other patches. The patches
> > that had nontrivial changes are the power and thermal ones involving
> > hwmon. I'm hoping to send up the patchset minus the hwmon patches in
> > the next version if there's no unforseen issues. If the hwmon patches
> > are ready then also, that's great, but otherwise those patches don't
> > need to hold up all the rest of the patchset. How's that sound?
>
> Hi Alan
>
> Thanks for your time for reviewing this patchset.
>
> This sounds good to me. Only thing here is, I need to split the patch which
> updates documentation into 2 patches (to remove hwmon description in doc),
> but for sure, it should be very easy. :)
Yes that sounds good.
Thanks,
Alan
>
> Thanks
> Hao
>
> >
> > Alan
> >
> > > >
> > > > This patch adds virtualization support description for DFL based
> > > > FPGA devices (based on PCIe SRIOV), and introductions to new
> > > > interfaces added by new dfl private feature drivers.
> > > >
> > > > Signed-off-by: Xu Yilun <[email protected]>
> > > > Signed-off-by: Wu Hao <[email protected]>
> > >
> > > Acked-by: Alan Tull <[email protected]>
> > >
> > > Thanks,
> > > Alan