From: Stuart Yoder <stuart.yoder@freescale.com>
To: Tillmann Heidsieck <theidsieck@leenox.de>,
        "gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
        Jose Rivera <German.Rivera@freescale.com>,
        "Katz Itai" <itai.katz@freescale.com>
CC: "devel@driverdev.osuosl.org" <devel@driverdev.osuosl.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "agraf@suse.de" <agraf@suse.de>, "arnd@arndb.de" <arnd@arndb.de>
Subject: RE: [PATCH v2] staging: fsl-mc: add DPAA2 overview readme
Thread-Topic: [PATCH v2] staging: fsl-mc: add DPAA2 overview readme
Thread-Index: AQHQ1CUOugIgUH1h70ekIiJXWpxJsZ4HjL/g
Date: Wed, 12 Aug 2015 01:03:04 +0000
Message-ID: <CY1PR0301MB0748F35BB79C3A9A9B56221C877E0@CY1PR0301MB0748.namprd03.prod.outlook.com>
References: <1438909764-16338-1-git-send-email-stuart.yoder@freescale.com>
 <55C9D601.2040305@leenox.de>
In-Reply-To: <55C9D601.2040305@leenox.de>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Aug 2015 01:03:04.7053
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 710a03f5-10f6-4d38-9ff4-a80b81da590d
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0301MB1641
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id t7C13HIP014279
Content-Length: 20828
Lines: 498


> -----Original Message-----
> From: Tillmann Heidsieck [mailto:theidsieck@leenox.de]
> Sent: Tuesday, August 11, 2015 6:01 AM
> To: Yoder Stuart-B08248; gregkh@linuxfoundation.org; Rivera Jose-B46482; katz Itai-RM05202
> Cc: devel@driverdev.osuosl.org; linux-kernel@vger.kernel.org; agraf@suse.de; arnd@arndb.de
> Subject: Re: [PATCH v2] staging: fsl-mc: add DPAA2 overview readme
> 
> Hi Stuart,
> 
> I am by no means a native speaker, but I have proof-read my fair share of articles and theses, so here are a
> bunch of suggestions which might or might not help improve you document.
> 
> On 07.08.2015 03:09, Stuart Yoder wrote:
> > add README file providing an overview of the DPAA2 architecture
> > and how it is integrated in Linux
> >
> > Signed-off-by: Stuart Yoder <stuart.yoder@freescale.com>
> > ---
> > -v2: added changelog text
> >
> >  drivers/staging/fsl-mc/README.txt | 364 ++++++++++++++++++++++++++++++++++++++
> >  drivers/staging/fsl-mc/TODO       |   4 -
> >  2 files changed, 364 insertions(+), 4 deletions(-)
> >  create mode 100644 drivers/staging/fsl-mc/README.txt
> >
> > diff --git a/drivers/staging/fsl-mc/README.txt b/drivers/staging/fsl-mc/README.txt
> > new file mode 100644
> > index 0000000..8214102
> > --- /dev/null
> > +++ b/drivers/staging/fsl-mc/README.txt
> > @@ -0,0 +1,364 @@
> > +Copyright (C) 2015 Freescale Semiconductor Inc.
> > +
> > +DPAA2 (Data Path Acceleration Architecture Gen2)
> > +------------------------------------------------
> > +
> > +This document provides an overview of the Freescale DPAA2 architecture
> > +and how it is integrated into the Linux kernel.
> > +
> > +Contents summary
> > +   -DPAA2 overview
> > +   -Overview of DPAA2 objects
> > +   -DPAA2 Linux driver architecture overview
> > +        -bus driver
> > +        -dprc driver
> 
> - DPRC driver
> 
> > +        -allocator
> > +        -dpio driver
> 
> - DPIO driver
> 
> > +        -Ethernet
> > +        -mac
> mac -> MAC
> > +
> > +DPAA2 Overview
> > +--------------
> > +
> > +DPAA2 is a hardware architecture designed for high-speeed network
> > +packet processing.  DPAA2 consists of sophisticated mechanisms for
> > +processing Ethernet packets, queue management, buffer management,
> > +autonomous L2 switching, virtual Ethernet bridging, and accelerator
> > +(e.g. crypto) sharing.
> > +
> > +A DPAA2 hardware component called the Management Complex (or MC) manages the
> > +DPAA2 hardware resources.  The MC provides an object-based abstraction for
> > +software drivers to use the DPAA2 hardware.
> > +
> > +The MC uses DPAA2 hardware resources such as queues, buffer pools, and
> > +network ports to create functional objects/devices such as network
> > +interfaces, an L2 switch, or accelerator instances.
> > +
> > +The MC provides memory-mapped I/O command interfaces (MC portals)
> > +which DPAA2 software drivers use to operate on DPAA2 objects:
> > +
> > +         +--------------------------------------+
> > +         |                  OS                  |
> > +         |                        DPAA2 drivers |
> > +         |                             |        |
> > +         +-----------------------------|--------+
> > +                                       |
> > +                                       | (create,discover,connect
> > +                                       |  config,use,destroy)
> > +                                       |
> > +                         DPAA2         |
> > +         +------------------------| mc portal |-+
> > +         |                             |        |
> > +         |   +- - - - - - - - - - - - -V- - -+  |
> > +         |   |                               |  |
> > +         |   |   Management Complex (MC)     |  |
> > +         |   |                               |  |
> > +         |   +- - - - - - - - - - - - - - - -+  |
> > +         |                                      |
> > +         | Hardware                  Hardware   |
> > +         | Resources                 Objects    |
> > +         | ---------                 -------    |
> > +         | -queues                   -DPRC      |
> > +         | -buffer pools             -DPMCP     |
> > +         | -Eth MACs/ports           -DPIO      |
> > +         | -network interface        -DPNI      |
> > +         |  profiles                 -DPMAC     |
> > +         | -queue portals            -DPBP      |
> > +         | -MC portals                ...       |
> > +         |  ...                                 |
> > +         |                                      |
> > +         +--------------------------------------+
> > +
> > +The MC mediates operations such as create, discover,
> > +connect, configuration, and destroy.  Fast-path operations
> > +on data, such as packet transmit/receive, are not mediated by
> > +the MC and are done directly using memory mapped regions in
> > +DPIO objects.
> > +
> > +Overview of DPAA2 Objects
> > +-------------------------
> > +The section provides a brief overview of some key objects
> > +in the DPAA2 hardware.  A simple scenario is described illustrating
> > +the objects involved in creating a network interfaces.
> > +
> > +-DPRC (Datapath Resource Container)
> > +
> > +    A DPRC is an container object that holds all the other
> 
> ... is a container ...
> 
> > +    types of DPAA2 objects.  In the example diagram below there
> > +    are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC)
> > +    in the container.
> > +
> > +    +---------------------------------------------------------+
> > +    | DPRC                                                    |
> > +    |                                                         |
> > +    |  +-------+  +-------+  +-------+  +-------+  +-------+  |
> > +    |  | DPMCP |  | DPIO  |  | DPBP  |  | DPNI  |  | DPMAC |  |
> > +    |  +-------+  +-------+  +-------+  +---+---+  +---+---+  |
> > +    |  | DPMCP |  | DPIO  |                                   |
> > +    |  +-------+  +-------+                                   |
> > +    |  | DPMCP |                                              |
> > +    |  +-------+                                              |
> > +    |                                                         |
> > +    +---------------------------------------------------------+
> > +
> > +    From the point of view of an OS, a DPRC is bus-like.  Like
> 
> maybe replace "is bus-like" with "behaves similar to a bus" or "can be viewed as a bus"
> 
> > +    a plug-and-play bus, such as PCI, DPRC commands can be used to
> 
> maybe replace "Like a pnp bus..." with "Similar to a plug-and-play bus, such as PCI, DPRC ..."
> 
> > +    enumerate the contents of the DPRC, discover the hardware
> > +    objects present (including mappable regions and interrupts).
> > +
> > +     dprc.1 (bus)
> > +       |
> > +       +--+--------+-------+-------+-------+
> > +          |        |       |       |       |
> > +        dpmcp.1  dpio.1  dpbp.1  dpni.1  dpmac.1
> > +        dpmcp.2  dpio.2
> > +        dpmcp.3
> > +
> > +    Hardware objects can be created and destroyed dynamically, providing
> > +    the ability to hot plug/unplug objects in and out of the DPRC.
> > +
> > +    A DPRC has a mappable mmio region (an MC portal) that can be used
> 
> - mmio -> MMIO (not sure whether mappable MMIO is redundant or not)
> - a MC portal
> 
> > +    to send MC commands.  It has an interrupt for status events (like
> > +    hotplug).
> > +
> > +    All objects in a container share the same hardware "isolation context".
> > +    This means that with respect to an IOMMU the isolation granularity
> > +    is at the DPRC (container) level, not at the individual object
> > +    level.
> > +
> > +    DPRCs can be defined statically and populated with objects
> > +    via a config file passed to the MC when firmware starts
> > +    it.  There is also a Linux user space tool called "restool"
> > +    that can be used to create/destroy containers and objects
> > +    dynamically.
> > +
> > +-DPAA2 Objects for an Ethernet Network Interface
> > +
> > +    A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX
> > +    queuing mechanisms, configuration mechanisms, buffer management,
> > +    physical ports, and interrupts.  DPAA2 uses a more granular approach
> > +    utilizing multiple hardware objects.  Each object has specialized
> > +    functions, and are used together by software to provide Ethernet network
> 
> Each object provides specialized functions. Groups of these objects are used by the software to provide
> Ethernet network interface functionality.
> 
> > +    interface functionality.  This approach provides efficient use of finite
> > +    hardware resources, flexibility, and performance advantages.
> > +
> > +    The diagram below shows the objects needed for a simple
> > +    network interface configuration on a system with 2 CPUs.
> > +
> > +              +---+---+ +---+---+
> > +                 CPU0     CPU1
> > +              +---+---+ +---+---+
> > +                  |         |
> > +              +---+---+ +---+---+
> > +                 DPIO     DPIO
> > +              +---+---+ +---+---+
> > +                    \     /
> > +                     \   /
> > +                      \ /
> > +                   +---+---+
> > +                      DPNI  --- DPBP,DPMCP
> > +                   +---+---+
> > +                       |
> > +                       |
> > +                   +---+---+
> > +                     DPMAC
> > +                   +---+---+
> > +                       |
> > +                    port/PHY
> > +
> > +    Below the objects are described.  For each object a brief description
> 
> The objects depicted in this figure are described below.
> 
> > +    is provided along with a summary of the kinds of operations the object
> > +    supports and a summary of key resources of the object (mmio regions
> > +    and irqs).
> 
> mmio -> MMIO
> irqs -> IRQs
> 
> > +
> > +       -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a
> > +        hardware device that connects to an Ethernet PHY and allows
> > +        physical transmission and reception of Ethernet frames.
> > +           -mmio regions: none
> 
> mmio -> MMIO
> 
> > +           -irqs: dpni link change
> 
> irqs -> IRQs
> dpni -> DPNI
> 
> > +           -commands: set link up/down, link config, get stats,
> > +             irq config, enable, reset
> 
> irq -> IRQ
> 
> > +
> > +       -DPNI (Datapath Network Interface): contains TX/RX queues,
> > +        network interface configuration, and rx buffer pool configuration
> 
> rx -> RX
> 
> > +        mechanisms.
> > +           -mmio regions: none
> 
> mmio -> MMIO
> 
> > +           -irqs: link state
> 
> irqs -> IRQs
> 
> > +           -commands: port config, offload config, queue config,
> > +            parse/classify config, irq config, enable, reset
> 
> irq -> IRQ
> 
> > +
> > +       -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue
> > +        packets and do hardware buffer pool management operations.  For
> > +        optimum performance there is typically DPIO per CPU.  This allows
> 
> For optimum performance there is typically one DPIO assigned to each CPU
> 
> > +        each CPU to perform simultaneous enqueue/dequeue operations.
> 
> This allows different CPUs to simultaneously perform enqueue/dequeue operations.
> 
> > +           -mmio regions: queue operations, buffer mgmt
> 
> mmio -> MMIO
> mgmt -> management
> 
> > +           -irqs: data availability, congestion notification, buffer
> > +                  pool depletion
> 
> irqs -> IRQs
> 
> > +           -commands: irq config, enable, reset
> 
> irq -> IRQ
> 
> > +
> > +       -DPBP (Datapath Buffer Pool): represents a hardware buffer
> > +        pool.
> > +           -mmio regions: none
> > +           -irqs: none
> > +           -commands: enable, reset
> 
> mmio -> MMIO
> irqs -> IRQs
> 
> > +
> > +       -DPMCP (Datapath MC Portal): provides an MC command portal.
> > +        Used by drivers to send commands to the MC to manage
> > +        objects.
> > +           -mmio regions: MC command portal
> > +           -irqs: command completion
> > +           -commands: irq config, enable, reset
> 
> mmio -> MMIO
> irqs -> IRQs
> irq -> IRQ
> 
> > +
> > +    Object Connections
> > +    ------------------
> > +    Some objects have explicit relationships that must
> > +    be configured:
> > +
> > +       -DPNI <--> DPMAC
> > +       -DPNI <--> DPNI
> > +       -DPNI <--> L2-switch-port
> > +          A DPNI must be connected to something such as a DPMAC,
> > +          another DPNI, or L2 switch port.  The DPNI connection
> > +          is made via a DPRC command.
> > +
> > +              +-------+  +-------+
> > +              | DPNI  |  | DPMAC |
> > +              +---+---+  +---+---+
> > +                  |          |
> > +                  +==========+
> > +
> > +       -DPNI <--> DPBP
> > +          A network interface requires a 'buffer pool' (DPBP
> > +          object) which provides a list of pointers to memory
> > +          where received Ethernet data is to be copied.  The
> > +          Ethernet driver configures the DPBPs associated with
> > +          the network interface.
> > +
> > +    Interrupts
> > +    ----------
> > +    All interrupts generated by DPAA2 objects are message
> > +    interrupts.  At the hardware level message interrupts
> > +    generated by devices will normally have 3 components--
> > +    1) a non-spoofable 'device-id' expressed on the hardware
> > +    bus, 2) an address, 3) a data value.
> > +
> > +    In the case of DPAA2 devices/objects, all objects in the
> > +    same container/DPRC share the same 'device-id'.
> > +    For ARM-based SoC this is the same as the stream ID.
> > +
> > +
> > +DPAA2 Linux Driver Overview
> > +---------------------------
> > +
> > +This section provides an overview of the Linux kernel drivers for
> > +DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure"
> > +drivers and 2) functional object drivers (such as Ethernet).
> > +
> > +As described previously, a DPRC is a container that holds the other
> > +types of DPAA2 objects.  It is functionally similar to a plug-and-play
> > +bus controller.
> > +
> > +Each object in the DPRC is a Linux "device" and is bound to a driver.
> > +The diagram below shows the Linux drivers involved in a networking
> > +scenario and the objects bound to each driver.  A brief description
> > +of each driver follows.
> > +
> > +                                             +------------+
> > +                                             | OS Network |
> > +                                             |   Stack    |
> > +                 +------------+              +------------+
> > +                 | Allocator  |. . . . . . . |  Ethernet  |
> > +                 |(dpmcp,dpbp)|              |   (dpni)   |
> > +                 +-.----------+              +---+---+----+
> > +                  .          .                   ^   |
> > +                 .            .     <data avail, |   |<enqueue,
> > +                .              .     tx confirm> |   | dequeue>
> > +    +-------------+             .                |   |
> > +    | DPRC driver |              .           +---+---V----+     +---------+
> > +    |   (dprc)    |               . . . . . .| DPIO driver|     |   MAC   |
> > +    +----------+--+                          |  (dpio)    |     | (dpmac) |
> > +               |                             +------+-----+     +-----+---+
> > +               |<dev add/remove>                    |                 |
> > +               |                                    |                 |
> > +          +----+--------------+                     |              +--+---+
> > +          |   mc-bus driver   |                     |              | PHY  |
> > +          |                   |                     |              |driver|
> > +          | /fsl-mc@80c000000 |                     |              +--+---+
> > +          +-------------------+                     |                 |
> > +                                                    |                 |
> > + ================================ HARDWARE =========|=================|======
> > +                                                  DPIO                |
> > +                                                    |                 |
> > +                                                  DPNI---DPBP         |
> > +                                                    |                 |
> > +                                                  DPMAC               |
> > +                                                    |                 |
> > +                                                   PHY ---------------+
> > + ===================================================|========================
> > +
> > +A brief description of each driver is provided below.
> > +
> > +    mc-bus driver
> > +    -------------
> mc-bus -> MC-Bus or MC-bus
> > +    The mc-bus driver is a platform driver and is probed from an
> > +    "/fsl-mc@xxxx" node in the device tree passed in by boot firmware.
> > +    It is responsible for bootstrapping the DPAA2 kernel infrastructure.
> > +    Key functions include:
> > +       -registering a new bus type named "fsl-mc" with the kernel,
> > +        and implementing bus call-backs (e.g. match/uevent/dev_groups)
> > +       -implemeting APIs for DPAA2 driver registration and for device
> > +        add/remove
> > +       -creates an MSI irq domain
> irq -> IRQ
> > +       -do a device add of the 'root' DPRC device, which is needed
> > +        to bootstrap things
> > +
> > +    DPRC driver
> > +    -----------
> > +    The dprc-driver is bound DPRC objects and does runtime management
> > +    of a bus instance.  It performs the initial bus scan of the DPRC
> > +    and handles interrupts for container events such as hot plug.
> > +
> > +    Allocator
> > +    ----------
> > +    Certain objects such as DPMCP and DPBP are generic and fungible,
> > +    and are intended to be used by other drivers.  For example,
> > +    the DPAA2 Ethernet driver needs:
> > +       -DPMCPs to send MC commands, to configure network interfaces
> > +       -DPBPs for network buffer pools
> > +
> > +    The allocator driver registers for these allocatable object types
> > +    and those objects are bound to the allocator when the bus is probed.
> > +    The allocator maintains a pool of objects that are available for
> > +    allocation by other DPAA2 drivers.
> > +
> > +    DPIO driver
> > +    -----------
> > +    The DPIO driver is bound to DPIO objects and provides services that allow
> > +    other drivers such as the Ethernet driver to receive and transmit data.
> > +    Key services include:
> > +        -data availability notifications
> > +        -hardware queuing operations (enqueue and dequeue of data)
> > +        -hardware buffer pool management
> > +
> > +    There is typically one DPIO object per physical CPU for optimum
> > +    performance, allowing each CPU to simultaneously enqueue
> > +    and dequeue data.
> 
> allowing different CPUs to simultaneously ...
> 
> > +
> > +    The DPIO driver operates on behalf of all DPAA2 drivers
> > +    active in the kernel--  Ethernet, crypto, compression,
> > +    etc.
> > +
> > +    Ethernet
> > +    --------
> > +    The Ethernet driver is bound to a DPNI and implements the kernel
> > +    interfaces needed to connect the DPAA2 network interface to
> > +    the network stack.
> > +
> > +    Each DPNI corresponds to a Linux network interface.
> > +
> > +    MAC driver
> > +    ----------
> > +    An Ethernet PHY is an off-chip, board specific component and is managed
> > +    by the appropriate PHY driver via an mdio bus.  The MAC driver
> > +    plays a role of being a proxy between the PHY driver and the
> > +    MC.  It does this proxy via the MC commands to a DPMAC object.
> > diff --git a/drivers/staging/fsl-mc/TODO b/drivers/staging/fsl-mc/TODO
> > index c29516b..3894368 100644
> > --- a/drivers/staging/fsl-mc/TODO
> > +++ b/drivers/staging/fsl-mc/TODO
> > @@ -1,7 +1,3 @@
> > -* Add README file (with ASCII art) describing relationships between
> > -  DPAA2 objects and how combine them to make a NIC, an LS2 switch, etc.
> > -  Also, define all acronyms used.
> > -
> >  * Decide if multiple root fsl-mc buses will be supported per Linux instance,
> >    and if so add support for this.
> >

Appreciate the detailed review.  Like most of your suggestions and
will incorporate on the next update to this document.

Stuart
????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay?ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m????????????I?