Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754589AbbHIOZ3 (ORCPT ); Sun, 9 Aug 2015 10:25:29 -0400 Received: from mx2.suse.de ([195.135.220.15]:41838 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754340AbbHIOZ1 (ORCPT ); Sun, 9 Aug 2015 10:25:27 -0400 Subject: Re: [PATCH v2] staging: fsl-mc: add DPAA2 overview readme To: Stuart Yoder , gregkh@linuxfoundation.org, german.rivera@freescale.com, itai.katz@freescale.com References: <1438909764-16338-1-git-send-email-stuart.yoder@freescale.com> Cc: devel@driverdev.osuosl.org, linux-kernel@vger.kernel.org, arnd@arndb.de From: Alexander Graf X-Enigmail-Draft-Status: N1110 Message-ID: <55C762D3.9070205@suse.de> Date: Sun, 9 Aug 2015 16:25:23 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <1438909764-16338-1-git-send-email-stuart.yoder@freescale.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 18963 Lines: 438 On 07.08.15 03:09, Stuart Yoder wrote: > add README file providing an overview of the DPAA2 architecture > and how it is integrated in Linux > > Signed-off-by: Stuart Yoder > --- > -v2: added changelog text > > drivers/staging/fsl-mc/README.txt | 364 ++++++++++++++++++++++++++++++++++++++ > drivers/staging/fsl-mc/TODO | 4 - > 2 files changed, 364 insertions(+), 4 deletions(-) > create mode 100644 drivers/staging/fsl-mc/README.txt > > diff --git a/drivers/staging/fsl-mc/README.txt b/drivers/staging/fsl-mc/README.txt > new file mode 100644 > index 0000000..8214102 > --- /dev/null > +++ b/drivers/staging/fsl-mc/README.txt > @@ -0,0 +1,364 @@ > +Copyright (C) 2015 Freescale Semiconductor Inc. > + > +DPAA2 (Data Path Acceleration Architecture Gen2) > +------------------------------------------------ > + > +This document provides an overview of the Freescale DPAA2 architecture > +and how it is integrated into the Linux kernel. > + > +Contents summary > + -DPAA2 overview > + -Overview of DPAA2 objects > + -DPAA2 Linux driver architecture overview > + -bus driver > + -dprc driver > + -allocator > + -dpio driver > + -Ethernet > + -mac > + > +DPAA2 Overview > +-------------- > + > +DPAA2 is a hardware architecture designed for high-speeed network > +packet processing. DPAA2 consists of sophisticated mechanisms for > +processing Ethernet packets, queue management, buffer management, > +autonomous L2 switching, virtual Ethernet bridging, and accelerator > +(e.g. crypto) sharing. > + > +A DPAA2 hardware component called the Management Complex (or MC) manages the > +DPAA2 hardware resources. The MC provides an object-based abstraction for > +software drivers to use the DPAA2 hardware. > + > +The MC uses DPAA2 hardware resources such as queues, buffer pools, and > +network ports to create functional objects/devices such as network > +interfaces, an L2 switch, or accelerator instances. > + > +The MC provides memory-mapped I/O command interfaces (MC portals) > +which DPAA2 software drivers use to operate on DPAA2 objects: > + > + +--------------------------------------+ > + | OS | > + | DPAA2 drivers | > + | | | > + +-----------------------------|--------+ > + | > + | (create,discover,connect > + | config,use,destroy) > + | > + DPAA2 | > + +------------------------| mc portal |-+ > + | | | > + | +- - - - - - - - - - - - -V- - -+ | > + | | | | > + | | Management Complex (MC) | | > + | | | | > + | +- - - - - - - - - - - - - - - -+ | > + | | > + | Hardware Hardware | > + | Resources Objects | > + | --------- ------- | > + | -queues -DPRC | > + | -buffer pools -DPMCP | > + | -Eth MACs/ports -DPIO | > + | -network interface -DPNI | > + | profiles -DPMAC | > + | -queue portals -DPBP | > + | -MC portals ... | > + | ... | > + | | > + +--------------------------------------+ > + > +The MC mediates operations such as create, discover, > +connect, configuration, and destroy. Fast-path operations > +on data, such as packet transmit/receive, are not mediated by > +the MC and are done directly using memory mapped regions in > +DPIO objects. > + > +Overview of DPAA2 Objects > +------------------------- > +The section provides a brief overview of some key objects > +in the DPAA2 hardware. A simple scenario is described illustrating > +the objects involved in creating a network interfaces. > + > +-DPRC (Datapath Resource Container) > + > + A DPRC is an container object that holds all the other > + types of DPAA2 objects. In the example diagram below there > + are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) > + in the container. > + > + +---------------------------------------------------------+ > + | DPRC | > + | | > + | +-------+ +-------+ +-------+ +-------+ +-------+ | > + | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | > + | +-------+ +-------+ +-------+ +---+---+ +---+---+ | > + | | DPMCP | | DPIO | | > + | +-------+ +-------+ | > + | | DPMCP | | > + | +-------+ | > + | | > + +---------------------------------------------------------+ > + > + From the point of view of an OS, a DPRC is bus-like. Like > + a plug-and-play bus, such as PCI, DPRC commands can be used to > + enumerate the contents of the DPRC, discover the hardware > + objects present (including mappable regions and interrupts). > + > + dprc.1 (bus) > + | > + +--+--------+-------+-------+-------+ > + | | | | | > + dpmcp.1 dpio.1 dpbp.1 dpni.1 dpmac.1 > + dpmcp.2 dpio.2 > + dpmcp.3 > + > + Hardware objects can be created and destroyed dynamically, providing > + the ability to hot plug/unplug objects in and out of the DPRC. > + > + A DPRC has a mappable mmio region (an MC portal) that can be used > + to send MC commands. It has an interrupt for status events (like > + hotplug). > + > + All objects in a container share the same hardware "isolation context". > + This means that with respect to an IOMMU the isolation granularity > + is at the DPRC (container) level, not at the individual object > + level. > + > + DPRCs can be defined statically and populated with objects > + via a config file passed to the MC when firmware starts > + it. There is also a Linux user space tool called "restool" > + that can be used to create/destroy containers and objects > + dynamically. Is this tool publicly available yet? Also, I find the naming unfortunate for a tool that potentially will get included in general purpose distributions. Naming it "dpaa2-restool" for example would make it much clearer what its purpose is and would give you a nice namespace to add more tools to later. > + > +-DPAA2 Objects for an Ethernet Network Interface > + > + A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX > + queuing mechanisms, configuration mechanisms, buffer management, > + physical ports, and interrupts. DPAA2 uses a more granular approach > + utilizing multiple hardware objects. Each object has specialized > + functions, and are used together by software to provide Ethernet network > + interface functionality. This approach provides efficient use of finite > + hardware resources, flexibility, and performance advantages. > + > + The diagram below shows the objects needed for a simple > + network interface configuration on a system with 2 CPUs. > + > + +---+---+ +---+---+ > + CPU0 CPU1 > + +---+---+ +---+---+ > + | | > + +---+---+ +---+---+ > + DPIO DPIO > + +---+---+ +---+---+ > + \ / > + \ / > + \ / > + +---+---+ > + DPNI --- DPBP,DPMCP > + +---+---+ > + | > + | > + +---+---+ > + DPMAC > + +---+---+ > + | > + port/PHY > + > + Below the objects are described. For each object a brief description > + is provided along with a summary of the kinds of operations the object > + supports and a summary of key resources of the object (mmio regions > + and irqs). > + > + -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a > + hardware device that connects to an Ethernet PHY and allows > + physical transmission and reception of Ethernet frames. > + -mmio regions: none > + -irqs: dpni link change > + -commands: set link up/down, link config, get stats, > + irq config, enable, reset > + > + -DPNI (Datapath Network Interface): contains TX/RX queues, > + network interface configuration, and rx buffer pool configuration > + mechanisms. > + -mmio regions: none > + -irqs: link state > + -commands: port config, offload config, queue config, > + parse/classify config, irq config, enable, reset > + > + -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue > + packets and do hardware buffer pool management operations. For I think you may want to explain the difference between TX/RX queues and "interfaces to enqueue and dequeue packets" ;). > + optimum performance there is typically DPIO per CPU. This allows typically [one] DPIO? > + each CPU to perform simultaneous enqueue/dequeue operations. > + -mmio regions: queue operations, buffer mgmt > + -irqs: data availability, congestion notification, buffer > + pool depletion > + -commands: irq config, enable, reset > + > + -DPBP (Datapath Buffer Pool): represents a hardware buffer > + pool. > + -mmio regions: none > + -irqs: none > + -commands: enable, reset > + > + -DPMCP (Datapath MC Portal): provides an MC command portal. > + Used by drivers to send commands to the MC to manage > + objects. > + -mmio regions: MC command portal > + -irqs: command completion > + -commands: irq config, enable, reset > + > + Object Connections > + ------------------ > + Some objects have explicit relationships that must > + be configured: > + > + -DPNI <--> DPMAC > + -DPNI <--> DPNI > + -DPNI <--> L2-switch-port > + A DPNI must be connected to something such as a DPMAC, > + another DPNI, or L2 switch port. The DPNI connection > + is made via a DPRC command. > + > + +-------+ +-------+ > + | DPNI | | DPMAC | > + +---+---+ +---+---+ > + | | > + +==========+ > + > + -DPNI <--> DPBP > + A network interface requires a 'buffer pool' (DPBP > + object) which provides a list of pointers to memory > + where received Ethernet data is to be copied. The > + Ethernet driver configures the DPBPs associated with > + the network interface. > + > + Interrupts > + ---------- > + All interrupts generated by DPAA2 objects are message > + interrupts. At the hardware level message interrupts > + generated by devices will normally have 3 components-- > + 1) a non-spoofable 'device-id' expressed on the hardware > + bus, 2) an address, 3) a data value. > + > + In the case of DPAA2 devices/objects, all objects in the > + same container/DPRC share the same 'device-id'. > + For ARM-based SoC this is the same as the stream ID. > + > + > +DPAA2 Linux Driver Overview > +--------------------------- > + > +This section provides an overview of the Linux kernel drivers for > +DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" > +drivers and 2) functional object drivers (such as Ethernet). > + > +As described previously, a DPRC is a container that holds the other > +types of DPAA2 objects. It is functionally similar to a plug-and-play > +bus controller. > + > +Each object in the DPRC is a Linux "device" and is bound to a driver. > +The diagram below shows the Linux drivers involved in a networking > +scenario and the objects bound to each driver. A brief description > +of each driver follows. > + > + +------------+ > + | OS Network | > + | Stack | > + +------------+ +------------+ > + | Allocator |. . . . . . . | Ethernet | > + |(dpmcp,dpbp)| | (dpni) | > + +-.----------+ +---+---+----+ > + . . ^ | > + . . + . . tx confirm> | | dequeue> > + +-------------+ . | | > + | DPRC driver | . +---+---V----+ +---------+ > + | (dprc) | . . . . . .| DPIO driver| | MAC | > + +----------+--+ | (dpio) | | (dpmac) | > + | +------+-----+ +-----+---+ > + | | | > + | | | > + +----+--------------+ | +--+---+ > + | mc-bus driver | | | PHY | > + | | | |driver| > + | /fsl-mc@80c000000 | | +--+---+ > + +-------------------+ | | > + | | > + ================================ HARDWARE =========|=================|====== > + DPIO | > + | | > + DPNI---DPBP | > + | | > + DPMAC | > + | | > + PHY ---------------+ > + ===================================================|======================== > + > +A brief description of each driver is provided below. > + > + mc-bus driver > + ------------- > + The mc-bus driver is a platform driver and is probed from an > + "/fsl-mc@xxxx" node in the device tree passed in by boot firmware. Probably better to describe the actual binding here which is based on compatible. > + It is responsible for bootstrapping the DPAA2 kernel infrastructure. > + Key functions include: > + -registering a new bus type named "fsl-mc" with the kernel, > + and implementing bus call-backs (e.g. match/uevent/dev_groups) > + -implemeting APIs for DPAA2 driver registration and for device implemeting? ;) > + add/remove > + -creates an MSI irq domain > + -do a device add of the 'root' DPRC device, which is needed > + to bootstrap things I think you can find a better way to describe exposure of the root container than "to bootstrap things". > + > + DPRC driver > + ----------- > + The dprc-driver is bound DPRC objects and does runtime management bound [to] DPRC > + of a bus instance. It performs the initial bus scan of the DPRC > + and handles interrupts for container events such as hot plug. > + > + Allocator > + ---------- > + Certain objects such as DPMCP and DPBP are generic and fungible, > + and are intended to be used by other drivers. For example, > + the DPAA2 Ethernet driver needs: > + -DPMCPs to send MC commands, to configure network interfaces > + -DPBPs for network buffer pools > + > + The allocator driver registers for these allocatable object types > + and those objects are bound to the allocator when the bus is probed. > + The allocator maintains a pool of objects that are available for > + allocation by other DPAA2 drivers. > + > + DPIO driver > + ----------- > + The DPIO driver is bound to DPIO objects and provides services that allow > + other drivers such as the Ethernet driver to receive and transmit data. > + Key services include: > + -data availability notifications > + -hardware queuing operations (enqueue and dequeue of data) > + -hardware buffer pool management > + > + There is typically one DPIO object per physical CPU for optimum > + performance, allowing each CPU to simultaneously enqueue > + and dequeue data. > + > + The DPIO driver operates on behalf of all DPAA2 drivers > + active in the kernel-- Ethernet, crypto, compression, > + etc. I'm not quite sure what this means? Where do I MMIO into when I want to transmit a packet? Alex > + > + Ethernet > + -------- > + The Ethernet driver is bound to a DPNI and implements the kernel > + interfaces needed to connect the DPAA2 network interface to > + the network stack. > + > + Each DPNI corresponds to a Linux network interface. > + > + MAC driver > + ---------- > + An Ethernet PHY is an off-chip, board specific component and is managed > + by the appropriate PHY driver via an mdio bus. The MAC driver > + plays a role of being a proxy between the PHY driver and the > + MC. It does this proxy via the MC commands to a DPMAC object. > diff --git a/drivers/staging/fsl-mc/TODO b/drivers/staging/fsl-mc/TODO > index c29516b..3894368 100644 > --- a/drivers/staging/fsl-mc/TODO > +++ b/drivers/staging/fsl-mc/TODO > @@ -1,7 +1,3 @@ > -* Add README file (with ASCII art) describing relationships between > - DPAA2 objects and how combine them to make a NIC, an LS2 switch, etc. > - Also, define all acronyms used. > - > * Decide if multiple root fsl-mc buses will be supported per Linux instance, > and if so add support for this. > > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/