Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Zhang, Tianfei" <tianfei.zhang@intel.com>
To:     "Wu, Hao" <hao.wu@intel.com>, "trix@redhat.com" <trix@redhat.com>,
        "mdf@kernel.org" <mdf@kernel.org>,
        "Xu, Yilun" <yilun.xu@intel.com>,
        "linux-fpga@vger.kernel.org" <linux-fpga@vger.kernel.org>,
        "linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
CC:     "corbet@lwn.net" <corbet@lwn.net>
Subject: RE: [PATCH v1 1/7] Documentation: fpga: dfl: add description of IOFS
Thread-Topic: [PATCH v1 1/7] Documentation: fpga: dfl: add description of IOFS
Thread-Index: AQHYIZYrI9khWBzXDEChcSjUIMBh1qyViSyAgAgTbdA=
Date:   Mon, 21 Feb 2022 07:39:35 +0000
Message-ID: <BN9PR11MB54830141CECE8076A7EBB1D9E33A9@BN9PR11MB5483.namprd11.prod.outlook.com>
References: <20220214112619.219761-1-tianfei.zhang@intel.com>
 <20220214112619.219761-2-tianfei.zhang@intel.com>
 <DM6PR11MB381933DF77F512D7E70539EF85359@DM6PR11MB3819.namprd11.prod.outlook.com>
In-Reply-To: <DM6PR11MB381933DF77F512D7E70539EF85359@DM6PR11MB3819.namprd11.prod.outlook.com>
Accept-Language: zh-CN, en-US
Content-Language: en-US
dlp-product: dlpe-windows
dlp-version: 11.6.200.16
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?meZrNuuQSklTirJW5Amzjgzyiw1FmCwAXRmxLj/G2PO2rjK+W12bdyyiqcuD?=
 =?us-ascii?Q?aAs08VI7ma3UqLk9BOE8CK2Ki+VeB8rKBDH4LTrgEfJ8Zhr8axH6AC2URlcH?=
 =?us-ascii?Q?Dwm7qL5SG2cXHOKp+jJn8HAdEUCyVRvBXYHQMOGgh//7EBbFq2zI8rRmLAZk?=
 =?us-ascii?Q?xCQodJigYi0qA9X4Ch1wouaHKj/iRCJAEJG0NpOukWnUBMdxLLdLBekTUAgs?=
 =?us-ascii?Q?maANxaakb4NI+2Fy+20C0393Uy7ODDSH26Bcf8hhI463Z7a+NQRxmP20eFc0?=
 =?us-ascii?Q?6FZCvTzqDyqYyISMrY+pznCTfcB8Grp76oMr/wK4o2bwkns4FntwNDFJM//Y?=
 =?us-ascii?Q?QBaPyBtOESlfj4B64KNvb5CEv2Mj6hLgzXGgtLsG9DWCAAYoudX4nB1/5BMk?=
 =?us-ascii?Q?am+SWP6trdE2IbUJJZ+KRWjzj1NTUOFjZDuEMdYzLDxzoI1NPngyZ581hEvL?=
 =?us-ascii?Q?kWXi6VoTQeTX1g3mOraaEPtlsYP0BNexGPkbOWRx1fmxW/9bdinWiezSE/we?=
 =?us-ascii?Q?616OdKb84Rn01BILPywpFg6BNCuQb2mSdP1FWaMoFxopbXqxxqAkK54mBYB3?=
 =?us-ascii?Q?cI7s3XOf8OfZb9hKCPMzMCFw3CXahEQ4A12AVmR4WvIUsKmdmjR7bIFcnVDI?=
 =?us-ascii?Q?q11zLSrQpoxUdaUM7aUlutHiIIufJxIIQhwmm/moJhhb2VliBLc9+D8KxnY+?=
 =?us-ascii?Q?a3xPFTmvPYMcsAPRowLgU/rPPsDnk6sMe3J1St3pnpnb2BZNmnwYR2IfzKWl?=
 =?us-ascii?Q?0Zl3pKUK4StIo9PnEzAkZocJyZQ+/U8UxL3iwZmBWU/GExo12uzqtZHE1JjI?=
 =?us-ascii?Q?15Ec6yyHfrkOA0zJeu4r76GT8MX8L/w/nx7SOs0uQAaYrr3j1OFjhXzIjV73?=
 =?us-ascii?Q?151jWdV9D2o9yRyI+JnLA45IB906E2sW61/a0YtKlcZr/1/titv2KJGOsLeG?=
 =?us-ascii?Q?XMm8FvNSWxk6n4I4TmY3PnIDlBCCisNdPGFVtkTPxRjcfMe99TE7DEunkG62?=
 =?us-ascii?Q?gDfh4NdLKl96YJ5ZCA92++IwCQhXktM51Lu09A6vj8w8C8RHAxi6GahtOWdD?=
 =?us-ascii?Q?MDWmpVybSXmNW+QmWocmX2JREqvmEHC8hYNdLdmg63PiKAicmXoBHq4Hx1FM?=
 =?us-ascii?Q?mGwvSkQSUbXfqzzFucgwzbSdP5LoR3BQ9MiiRFaz89a4goDDI+grLlfl2/vJ?=
 =?us-ascii?Q?DlH4LGLpn8vYESHe5/ORmFj1f9Y7eckrXaOnSVOUq6s4t2sjgiPHZwXN2+e9?=
 =?us-ascii?Q?ZgD8Zr6YS7InPg9ARbD9g2ZK/QOzNoEMXJBloccDHbie9n60sbhG37io+l9f?=
 =?us-ascii?Q?afiEYPlIIzgR7n4etMamGr6yB0F8FhYWkBYltpKX8+okk538SvawbX4tZd4n?=
 =?us-ascii?Q?6blEv1CuwPG5y3LJayFUyqu7Mit1CnjlLCtfm9EERJdeLZO29VEOIYCHVxQN?=
 =?us-ascii?Q?NxHg0yatc4wwk/cfu6mtHkxg9f4ELJoiTmuZd8iB8my6VwkxzK2M7lJoiX+1?=
 =?us-ascii?Q?hu/L3PzIz4ofrs+wgpe9XgcFEnG8kpDC75tWtj9n8lLf1b8gMNHFLYNS+MNd?=
 =?us-ascii?Q?rLong22dOa9Zr5c6F4q0D3nztx/Lh3t4jHozza1OTR5NrBmV/RShcdBLyshK?=
 =?us-ascii?Q?0L25g+1w/RnE/jGruZVWghI=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5483.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c9cc3256-d4ba-43ee-6c41-08d9f50d4f62
X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Feb 2022 07:39:35.8928
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: FNYxSkvN2DdxD73SPCHG0Pmg/5XUhHm3jEPEbNtiLDMQDqKbgjNyUGBNUn1rFcbilujrzstNBJHdf2tBSOwcqg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR11MB4255
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> -----Original Message-----
> From: Wu, Hao <hao.wu@intel.com>
> Sent: Wednesday, February 16, 2022 11:35 AM
> To: Zhang, Tianfei <tianfei.zhang@intel.com>; trix@redhat.com;
> mdf@kernel.org; Xu, Yilun <yilun.xu@intel.com>; linux-fpga@vger.kernel.or=
g;
> linux-doc@vger.kernel.org; linux-kernel@vger.kernel.org
> Cc: corbet@lwn.net
> Subject: Re: [PATCH v1 1/7] Documentation: fpga: dfl: add description of =
IOFS
>=20
> > Subject: [PATCH v1 1/7] Documentation: fpga: dfl: add description of
> > IOFS
> >
> > From: Tianfei Zhang <tianfei.zhang@intel.com>
> >
> > This patch adds description about IOFS support for DFL.
> >
> > Signed-off-by: Tianfei Zhang <tianfei.zhang@intel.com>
> > ---
> >  Documentation/fpga/dfl.rst | 99
> > +++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 97 insertions(+), 2 deletions(-)
> >
> > diff --git a/Documentation/fpga/dfl.rst b/Documentation/fpga/dfl.rst
> > index ef9eec71f6f3..6f9eae1c1697 100644
> > --- a/Documentation/fpga/dfl.rst
> > +++ b/Documentation/fpga/dfl.rst
> > @@ -58,7 +58,10 @@ interface to FPGA, e.g. the FPGA Management Engine
> > (FME) and Port (more
> >  descriptions on FME and Port in later sections).
> >
> >  Accelerated Function Unit (AFU) represents an FPGA programmable
> > region and -always connects to a FIU (e.g. a Port) as its child as illu=
strated
> above.
> > +always connects to a FIU (e.g. a Port) as its child as illustrated
> > +above, but on IOFS design, it introducing Port Gasket which contains
> > +AFUs. For DFL
> > perspective,
> > +the Next_AFU pointer on FIU feature header can point to NULL so the
> > +AFU is
> > not
> > +connects to a FIU(more descriptions on IOFS in later section).
> >
> >  Private Features represent sub features of the FIU and AFU. They
> > could be  various function blocks with different IDs, but all private
> > features which @@ -134,6 +137,9 @@ reconfigurable region containing an
> > AFU. It controls the communication from SW  to the accelerator and
> > exposes features such as reset and debug. Each FPGA  device may have
> > more than one port, but always one AFU per port.
> >
> > +On IOFS, it introducing a new hardware unit, Port Gasket, which
> > +contains all the PR specific modules and regions (more descriptions on=
 IOFS in
> later section).
>=20
> What's the different between the PORT we have now for DFH, and the new on=
e
> in IOFS?

From DFL perspective , the PORT concept is identical between IOFS and old c=
ard like N3000 card.
The major different is that the IOFS introducing a new hardware concept: Po=
rt Gasket, which include the PR slot,=20
Port control, Port user clock control and Port errors.
>=20
> > +
> >
> >  AFU
> >  =3D=3D=3D
> > @@ -143,6 +149,9 @@ used for accelerator-specific control registers.
> >  User-space applications can acquire exclusive access to an AFU
> > attached to a  port by using open() on the port device node and release=
 it using
> close().
> >
> > +On IOFS, the AFU is embedded in a Port Gasket. The AFU resource can
> > +expose
> > via
> > +VFs with SRIOV support (more descriptions on IOFS in later section).
> > +
> >  The following functions are exposed through ioctls:
> >
> >  - Get driver API version (DFL_FPGA_GET_API_VERSION) @@ -284,7 +293,8
> > @@ FME is always accessed through the physical function (PF).
> >
> >  Ports (and related AFUs) are accessed via PF by default, but could be
> > exposed  through virtual function (VF) devices via PCIe SRIOV. Each VF
> > only contains
> > -1 Port and 1 AFU for isolation. Users could assign individual VFs
> > (accelerators)
> > +1 Port (On IOFS design, the VF is designs without Port) and 1 AFU for
> isolation.
> > +Users could assign individual VFs (accelerators)
> >  created via PCIe SRIOV interface, to virtual machines.
> >
> >  The driver organization in virtualization case is illustrated below:
> > @@ -389,6 +399,91 @@ The device nodes used for ioctl() or mmap() can
> > be referenced through::
> >  	/sys/class/fpga_region/<regionX>/<dfl-fme.n>/dev
> >  	/sys/class/fpga_region/<regionX>/<dfl-port.n>/dev
> >
> > +Intel Open FPGA stack
> > +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > +Intel Open FPGA stack aka IOFS, Intel's version of a common core set
> > +of RTL to allow customers to easily interface to logic and IP on the F=
PGA.
> > +IOFS leverage the DFL for the implementation of the FPGA RTL design.
> > +
> > +IOFS designs allow for the arrangement of software interfaces across
> > +multiple PCIe endpoints. Some of these interfaces may be PFs defined
> > +in the static
> > region
> > +that connect to interfaces in an IP that is loaded via Partial
> > +Reconfiguration
> > (PR).
> > +And some of these interfaces may be VFs defined in the PR region that
> > +can be reconfigured by the end-user. Furthermore, these PFs/VFs may
> > +also be
> > arranged
> > +using a DFL such that features may be discovered and accessed in user
> > +space (with the aid of a generic kernel driver like vfio-pci). The
> > +diagram below depicts an example design with two PFs and two VFs. In
> > +this example, PF1 implements
> > its
> > +MMIO space such that it is compatible with the VirtIO framework. The
> > +other
> > functions,
> > +VF0 and VF1, leverage VFIO to export the MMIO space to an application
> > +or a
> > hypervisor.
>=20
> So PORT will never be exposed to VM in IOFS design, is my understanding
> correct?

On IOFS, there are many methods to access the AFU for virtualization and no=
n-virtualization usage.
1. Legacy Model. This is used for N3000 and N5000 card.
In this model the entire AFU region is a unit of PR, and there is a Port de=
vice connected to this AFU.
In this model, the follow of exposing to VM is the same we have in N3000 an=
d N5000 card.
The major follow is:
* release the port device.
* configure the SRIOV
* assign the VF to VM
* put the port device back to PF after finished access in VM
2. Micro-Personas in AFU.=20
IOFS intruding new model for PR and AFU access.
Micro-Personas allow the RTL developer to designate their own AFU-defined P=
R regions.=20
In this model, each PR slot has a dedicate  Port device, the follow of expo=
sing to VM is the same we have in N3000 and N5000 card.
3. Multiple VFs per PR slot.
In this method, we can instance multiple VFs over SRIOV for one PR slot, an=
d access the AFU resource by different VFs in virtualization usage.
 In this case, the Port device would not connected to AFU/PR slot, so we do=
n't need to release the Port device.
before we assign the VF to VM.

Which model will deploy in IOFS product depends on the RTL developer.

>=20
> > +
> > +     +-----------------+  +--------------+  +-------------+  +--------=
----+
> > +     | FPGA Managerment|  |   VirtIO     |  |  User App   |  | Virtual=
    |
>=20
> s/Managerment/Management/

Thanks, I will fix it on next version.

>=20
> > +     |      App        |  |     App      |  |             |  | Machine=
    |
> > +     +--------+--------+  +------+-------+  +------+------+  +-----+--=
----+
> > +              |                  |                 |               |
> > +              |                  |                 |               |
> > +     +--------+--------+  +------+-------+  +------+------+        |
> > +     |     DFL Driver  |  |VirtIO driver |  |    VFIO     |        |
> > +     +--------+--------+  +------+-------+  +------+------+        |
> > +              |                  |                 |               |
> > +              |                  |                 |               |
> > +     +--------+--------+  +------+-------+  +------+------+   +----+--=
----+
> > +     |     PF0         |  |     PF1      |  |   PF0_VF0   |   |  PF0_V=
F1  |
> > +     +-----------------+  +--------------+  +-------------+   +-------=
----+
> > +
> > +On IOFS, it introducing some enhancements compared with original DFL
> design.
> > +1. It introducing Port Gasket in PF0 which is responsible for FPGA
> > +management, like FME and Port management. The Port Gasket contains
> > +all the PR specific
> > modules
>=20
> So in IOFS, in PF0, we always have FME and PORT DFH, is my understanding
> correct?

Yes.

> Then why we need patch #3?

This is for " Multiple VFs per PR slot" model, in this model, the Port devi=
ce would not connected to AFU (the BarID of Port device should be set to in=
valid),
so we just can access PR slot/AFU resource via VFs.

>=20
> Another question is in IOFS, do we need to support multiple PR regions/Po=
rts?
> If that is the case, how should we know which VFs belongs to PORT1 or POR=
T2?
>=20
> > +and logic, e.g., PR slot reset/freeze control, user clock, remote STP =
etc.
> > +Architecturally, a Port Gasket can have multiple PR slots where user
> > +workload
> > can
> > +be programmed into.
> > +2. To expend the scalable of FPGA, it can support multiple FPs in
> > +static region
>=20
> s/FPs/PFs/

I will fix on next version.

>=20
> > +which contain some static functions like VirtIO, diagnostic test, and
> > +access over VFIO or assigned to VMs easily. Those PFs will not have a
> > +Port Unit which
> > without
> > +PR region (AFU) connected to those PFs, and the end-user cannot
> > +partial
> > reconfigurate
>=20
> s/reconfigurate/reconfigure/

I will fix on next version.

>=20
> > +those PFs.
> > +3. In our previous DFL design, it can only create one VF based in an
> > +AFU. To
> > raise
> > +the efficiency usage of AFU, it can create more than one VFs in an
> > +AFU via PCIe SRIOV, so those VFs share the PR region and resource.
> > +
> > +There is one reference architecture design for IOFS as illustrated bel=
ow:
> > +
> > +                              +----------------------+
> > +                              |   PF/VF mux/demux    |
> > +                              +--+--+-----+------+-+-+
> > +                                 |  |     |      | |
> > +        +------------------------+  |     |      | |
> > +  PF0   |                 +---------+   +-+      | |
> > +    +---+---+             |         +---+----+   | |
> > +    |  DFH  |             |         |   DFH  |   | |
> > +    +-------+       +-----+----+    +--------+   | |
> > +    |  FME  |       |  VirtIO  |    |  Test  |   | |
> > +    +-------+       +----------+    +--------+   | |
> > +    | Port  |            PF1            PF2      | |
> > +    +---+---+                                    | |
> > +        |                             +----------+ |
> > +        |                             |           ++
> > +        |                             |           |
> > +        |                             | PF0_VF0   | PF0_VF1
> > +        |           +-----------------+-----------+------------+
> > +        |           |           +-----+-----------+--------+   |
> > +        |           |           |     |           |        |   |
> > +        |           | +------+  |  +--+ -+     +--+---+    |   |
> > +        |           | | CSR  |  |  | DFH |     |  DFH |    |   |
> > +        +-----------+ +------+  |  +-----+     +------+    |   |
> > +                    |           |  | DEV |     |  DEV |    |   |
> > +                    |           |  +-----+     +------+    |   |
> > +                    |           |            PR Slot       |   |
> > +                    |           +--------------------------+   |
> > +                    | Port Gasket                              |
> > +                    +------------------------------------------+
> > +
> > +Here are the major changes about DFL structures on IOFS
> > +implementation
> > design:
> > +1. The Port Gasket connects to FIU Port in DFL, but the Next_AFU
> > +pointer in FIU feature header can point to NULL so that it is no AFU
> > +connects to a FIU Port.
> > +2. The VF which include in PR region can start with AFU feature
> > +header without a FIU Port feature header.
>=20
> What about PF2 in static region? Which type of DFH will be used?

An IP designer may choose to add more than one PF for interfacing with IP o=
n the FPGA.
If at least one PF implements a DFL with management features such as an FME=
 or PR, then
the device can be managed using the IOFS software stack. For example, a des=
ign may include
FME and PR on PF0 and the actual workload interfaces on PF1. If a workload=
=20
implements virtio-net backend and interface, the IOFS software stack will o=
nly bind to and
communicate with IOFS features/interfaces found in the DFL on PF0. The seco=
nd PF, PF1,=20
will bind with virtio-net driver presenting itself as a network interface t=
o the OS.
So the IOFS providing the diversity for IP designer.

>=20
> Thanks
> Hao
>=20
> >
> >  Performance Counters
> >  =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > --
> > 2.17.1