This creates a DMAengine driver for the ARM PL080/PL081 PrimeCells
based on the implementation earlier submitted by Peter Pearse.
This is working like a charm for memcpy on the PB11MPCore, but
slave DMA to devices is still not working.
This DMA controller is used in mostly unmodified form in the ARM
RealView and Versatile platforms, in the ST-Ericsson Nomadik, and
in the ST SPEAr platform.
It has been converted to use the header from the Samsung PL080
derivate instead of its own defintions, and can potentially support
several controllers in the same system.
Cc: Peter Pearse <[email protected]>
Cc: Ben Dooks <[email protected]>
Cc: Kukjin Kim <[email protected]>
Cc: Alessandro Rubini <[email protected]>
Cc: Viresh Kumar <[email protected]>
Signed-off-by: Linus Walleij <[email protected]>
---
arch/arm/include/asm/hardware/pl080.h | 2 +
drivers/dma/Kconfig | 9 +
drivers/dma/Makefile | 1 +
drivers/dma/amba-pl08x.c | 1925 +++++++++++++++++++++++++++++++++
include/linux/amba/pl08x.h | 173 +++
5 files changed, 2110 insertions(+), 0 deletions(-)
create mode 100644 drivers/dma/amba-pl08x.c
create mode 100644 include/linux/amba/pl08x.h
diff --git a/arch/arm/include/asm/hardware/pl080.h b/arch/arm/include/asm/hardware/pl080.h
index f70e1e9..f35b86e 100644
--- a/arch/arm/include/asm/hardware/pl080.h
+++ b/arch/arm/include/asm/hardware/pl080.h
@@ -68,6 +68,8 @@
#define PL080_CONTROL_TC_IRQ_EN (1 << 31)
#define PL080_CONTROL_PROT_MASK (0x7 << 28)
#define PL080_CONTROL_PROT_SHIFT (28)
+#define PL080_CONTROL_PROT_CACHE (1 << 30)
+#define PL080_CONTROL_PROT_BUFF (1 << 29)
#define PL080_CONTROL_PROT_SYS (1 << 28)
#define PL080_CONTROL_DST_INCR (1 << 27)
#define PL080_CONTROL_SRC_INCR (1 << 26)
diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig
index 8344375..f4655c2 100644
--- a/drivers/dma/Kconfig
+++ b/drivers/dma/Kconfig
@@ -36,6 +36,15 @@ comment "DMA Devices"
config ASYNC_TX_DISABLE_CHANNEL_SWITCH
bool
+config AMBA_PL08X
+ bool "ARM PrimeCell PL080 or PL81 support"
+ depends on ARM_AMBA && EXPERIMENTAL
+ default y if ARCH_REALVIEW
+ select DMA_ENGINE
+ help
+ Platform has a PL08x DMAC device
+ which can provide DMA engine support
+
config INTEL_IOATDMA
tristate "Intel I/OAT DMA support"
depends on PCI && X86
diff --git a/drivers/dma/Makefile b/drivers/dma/Makefile
index 0fe5ebb..c51dbc8 100644
--- a/drivers/dma/Makefile
+++ b/drivers/dma/Makefile
@@ -23,3 +23,4 @@ obj-$(CONFIG_AMCC_PPC440SPE_ADMA) += ppc4xx/
obj-$(CONFIG_TIMB_DMA) += timb_dma.o
obj-$(CONFIG_STE_DMA40) += ste_dma40.o ste_dma40_ll.o
obj-$(CONFIG_PL330_DMA) += pl330.o
+obj-$(CONFIG_AMBA_PL08X) += amba-pl08x.o
diff --git a/drivers/dma/amba-pl08x.c b/drivers/dma/amba-pl08x.c
new file mode 100644
index 0000000..50531fa
--- /dev/null
+++ b/drivers/dma/amba-pl08x.c
@@ -0,0 +1,1925 @@
+/*
+ * Copyright (c) 2006 ARM Ltd.
+ * Copyright (c) 2010 ST-Ericsson SA
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc., 59
+ * Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ * The full GNU General Public License is iin this distribution in the
+ * file called COPYING.
+ *
+ * Documentation: ARM DDI 0196G == PL080
+ * Documentation: ARM DDI 0218E == PL081
+ *
+ * PL080 & PL081 both have 16 sets of DMA signals. They differ in the number
+ * of channels which may be in use at once. Also PL080 has a dual bus master,
+ * PL081 has a single master.
+ *
+ * Memory to peripheral transfer may be visualized as
+ * Get data from memory to DMAC
+ * Until no data left
+ * On burst request from peripheral
+ * Destination burst from DMAC to peripheral
+ * Clear burst request
+ * Raise terminal count interrupt
+ *
+ * For peripherals with a FIFO:
+ * Source burst size == half the depth of the peripheral FIFO
+ * Destination burst size == width of the peripheral FIFO
+ *
+ * (Bursts are irrelevant for mem to mem transfers - there are no burst
+ * signals)
+ *
+ * ASSUMES default (little) endianness for DMA transfers
+ *
+ * Only DMAC flow control is implemented
+ *
+ * Global TODO:
+ * - Break out common code from arch/arm/mach-s3c64xx and share
+ */
+#include <linux/device.h>
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/interrupt.h>
+#include <linux/slab.h>
+#include <linux/dmapool.h>
+#include <linux/amba/bus.h>
+#include <linux/dmaengine.h>
+#include <linux/amba/pl08x.h>
+#include <linux/amba/dma.h>
+
+#include <asm/hardware/pl080.h>
+#include <asm/dma.h>
+#include <asm/mach/dma.h>
+#include <asm/atomic.h>
+#include <asm/processor.h>
+#include <asm/cacheflush.h>
+
+#define DRIVER_NAME "pl08xdmac"
+
+/**
+ * struct vendor_data - vendor-specific config parameters
+ * for PL08x derivates
+ * @name: the name of this specific variant
+ * @channels: the number of channels available in this variant
+ */
+struct vendor_data {
+ char *name;
+ u8 channels;
+};
+
+/*
+ * PL08X private data structures
+ * An LLI struct - see pl08x TRM
+ * Note that next uses bit[0] as a bus bit,
+ * start & end do not - their bus bit info
+ * is in cctl
+ */
+struct lli {
+ dma_addr_t src;
+ dma_addr_t dst;
+ dma_addr_t next;
+ u32 cctl;
+};
+
+/**
+ * struct pl08x_driver_data - the local state holder for the PL08x
+ * @base: virtual memory base (remapped) for the PL08x
+ * @adev: the corresponding AMBA (PrimeCell) bus entry
+ * @vd: vendor data for this PL08x variant
+ * @pd: platform data passed in from the platform/machine
+ * @phy_chans: array of data for the physical channels
+ * @pool: a pool for the LLI descriptors
+ * @pool_ctr: counter of LLIs in the pool
+ * @max_num_llis: maximum number of LLIs, i.e. longest linked transfer
+ * length, submitted so far
+ * @lock: a spinlock for this struct
+ */
+struct pl08x_driver_data {
+ void __iomem *base;
+ struct amba_device *adev;
+ struct vendor_data *vd;
+ struct pl08x_platform_data *pd;
+ struct pl08x_phy_chan *phy_chans;
+ struct dma_pool *pool;
+ int pool_ctr;
+ int max_num_llis;
+ spinlock_t lock;
+};
+
+#ifdef MODULE
+
+# error "AMBA PL08X DMA CANNOT BE COMPILED AS A LOADABLE MODULE AT PRESENT"
+
+/*
+ a) Some devices might make use of DMA during boot
+ (esp true for DMAENGINE implementation)
+ b) Memory allocation will need much more attention
+ before load/unload can be supported
+ */
+#endif
+
+/*
+ * PL08X specific defines
+ */
+
+/*
+ * Memory boundaries: the manual for PL08x says that the controller
+ * cannot read past a 1KiB boundary, so these defines are used to
+ * create transfer LLIs that do not cross such boundaries.
+ */
+#define PL08X_BOUNDARY_SHIFT (10) /* 1KB 0x400 */
+#define PL08X_BOUNDARY_SIZE (1 << PL08X_BOUNDARY_SHIFT)
+
+/* Minimum period between work queue runs */
+#define PL08X_WQ_PERIODMIN 20
+
+/* Size (bytes) of each LLI buffer allocated for one transfer */
+# define PL08X_LLI_TSFR_SIZE 0x2000
+
+/* Maximimum times we call dma_pool_alloc on this pool without freeing */
+#define PL08X_MAX_ALLOCS 0x40
+#define MAX_NUM_TSFR_LLIS (PL08X_LLI_TSFR_SIZE/sizeof(struct lli))
+#define PL08X_ALIGN 8
+
+static inline struct pl08x_dma_chan *to_pl08x_chan(struct dma_chan *chan)
+{
+ return container_of(chan, struct pl08x_dma_chan, chan);
+}
+
+/*
+ * Physical channel handling
+ */
+
+/* Whether a certain channel is busy or not */
+static int pl08x_phy_channel_busy(struct pl08x_phy_chan *ch)
+{
+ unsigned int val;
+
+ val = readl(ch->base + PL080_CH_CONFIG);
+ return val & PL080_CONFIG_ACTIVE;
+}
+
+/*
+ * Set the initial DMA register values i.e. those for the first LLI
+ * The next lli pointer and the configuration interrupt bit have
+ * been set when the LLIs were constructed
+ */
+static void pl08x_set_cregs(struct pl08x_driver_data *pl08x,
+ struct pl08x_phy_chan *ch)
+{
+ u32 val;
+
+ /* Wait for channel inactive */
+ val = readl(ch->base + PL080_CH_CONFIG);
+ while (val & PL080_CONFIG_ACTIVE)
+ val = readl(ch->base + PL080_CH_CONFIG);
+
+ dev_vdbg(&pl08x->adev->dev,
+ "WRITE channel %d: csrc=%08x, cdst=%08x, "
+ "cctl=%08x, clli=%08x, ccfg=%08x\n",
+ ch->id,
+ ch->csrc,
+ ch->cdst,
+ ch->cctl,
+ ch->clli,
+ ch->ccfg);
+
+ writel(ch->csrc, ch->base + PL080_CH_SRC_ADDR);
+ writel(ch->cdst, ch->base + PL080_CH_DST_ADDR);
+ writel(ch->clli, ch->base + PL080_CH_LLI);
+ writel(ch->cctl, ch->base + PL080_CH_CONTROL);
+ writel(ch->ccfg, ch->base + PL080_CH_CONFIG);
+ mb();
+}
+
+static inline void pl08x_config_phychan_for_txd(struct pl08x_dma_chan *plchan)
+{
+ struct pl08x_channel_data *cd = plchan->cd;
+ struct pl08x_phy_chan *phychan = plchan->phychan;
+ struct pl08x_txd *txd = plchan->at;
+
+ /* Copy the basic control register calculated at transfer config */
+ phychan->csrc = txd->csrc;
+ phychan->cdst = txd->cdst;
+ phychan->clli = txd->clli;
+ phychan->cctl = txd->cctl;
+
+ /* Assign the signal to the proper control registers */
+ phychan->ccfg = cd->ccfg;
+ phychan->ccfg &= ~PL080_CONFIG_SRC_SEL_MASK;
+ phychan->ccfg &= ~PL080_CONFIG_DST_SEL_MASK;
+ /* If it wasn't set from AMBA, ignore it */
+ if (txd->direction == DMA_TO_DEVICE)
+ /* Select signal as destination */
+ phychan->ccfg |=
+ (phychan->signal << PL080_CONFIG_DST_SEL_SHIFT);
+ else if (txd->direction == DMA_FROM_DEVICE)
+ /* Select signal as source */
+ phychan->ccfg |=
+ (phychan->signal << PL080_CONFIG_SRC_SEL_SHIFT);
+ /* Always enable error interrupts */
+ phychan->ccfg |= PL080_CONFIG_ERR_IRQ_MASK;
+ /* Always enable terminal interrupts */
+ phychan->ccfg |= PL080_CONFIG_TC_IRQ_MASK;
+}
+
+/*
+ * Enable the DMA channel
+ * Assumes all other configuration bits have been set
+ * as desired before this code is called
+ */
+static void pl08x_enable_phy_chan(struct pl08x_driver_data *pl08x,
+ struct pl08x_phy_chan *ch)
+{
+ u32 val;
+
+ /*
+ * Do not access config register until channel shows as disabled
+ */
+ while (readl(pl08x->base + PL080_EN_CHAN) & (1 << ch->id))
+ ;
+
+ /*
+ * Do not access config register until channel shows as inactive
+ */
+ val = readl(ch->base + PL080_CH_CONFIG);
+ while ((val & PL080_CONFIG_ACTIVE) || (val & PL080_CONFIG_ENABLE))
+ val = readl(ch->base + PL080_CH_CONFIG);
+
+ writel(val | PL080_CONFIG_ENABLE, ch->base + PL080_CH_CONFIG);
+ mb();
+}
+
+/*
+ * Overall DMAC remains enabled always.
+ *
+ * Disabling individual channels could lose data.
+ *
+ * Disable the peripheral DMA after disabling the DMAC
+ * in order to allow the DMAC FIFO to drain, and
+ * hence allow the channel to show inactive
+ *
+ */
+static void pl08x_pause_phy_chan(struct pl08x_phy_chan *ch)
+{
+ u32 val;
+
+ /* Set the HALT bit and wait for the FIFO to drain */
+ val = readl(ch->base + PL080_CH_CONFIG);
+ val |= PL080_CONFIG_HALT;
+ writel(val, ch->base + PL080_CH_CONFIG);
+
+ /* Wait for channel inactive */
+ val = readl(ch->base + PL080_CH_CONFIG);
+ while (val & PL080_CONFIG_ACTIVE)
+ val = readl(ch->base + PL080_CH_CONFIG);
+
+ mb();
+
+ return;
+}
+
+static void pl08x_resume_phy_chan(struct pl08x_phy_chan *ch)
+{
+ u32 val;
+
+ /* Clear the HALT bit */
+ val = readl(ch->base + PL080_CH_CONFIG);
+ val &= ~PL080_CONFIG_HALT;
+ writel(val, ch->base + PL080_CH_CONFIG);
+ mb();
+
+ return;
+}
+
+
+/* Stops the channel */
+static void pl08x_stop_phy_chan(struct pl08x_phy_chan *ch)
+{
+ u32 val;
+
+ pl08x_pause_phy_chan(ch);
+
+ /* Disable channel */
+ val = readl(ch->base + PL080_CH_CONFIG);
+ val &= ~PL080_CONFIG_ENABLE;
+ writel(val, ch->base + PL080_CH_CONFIG);
+ mb();
+
+ return;
+}
+
+static inline u32 get_bytes_in_cctl(u32 cctl)
+{
+ /* The source width defines the number of bytes */
+ u32 bytes = cctl & PL080_CONTROL_TRANSFER_SIZE_MASK;
+
+ switch ((cctl >> 18) & 3) {
+ case PL080_WIDTH_8BIT:
+ break;
+ case PL080_WIDTH_16BIT:
+ bytes *= 2;
+ break;
+ case PL080_WIDTH_32BIT:
+ bytes *= 4;
+ break;
+ }
+ return bytes;
+}
+
+static u32 pl08x_getbytes_phy_chan(struct pl08x_phy_chan *ch)
+{
+ u32 bytes;
+
+ /* FIXME: follow all queued transactions */
+ bytes = get_bytes_in_cctl(readl(ch->base + PL080_CH_CONTROL));
+ /* TODO: follow the LLI to see the sum summarum */
+ return bytes;
+}
+
+/*
+ * Allocate a physical channel for a virtual channel
+ */
+static struct pl08x_phy_chan *
+pl08x_get_phy_channel(struct pl08x_driver_data *pl08x,
+ struct pl08x_dma_chan *virt_chan)
+{
+ struct pl08x_phy_chan *ch = NULL;
+ unsigned long flags;
+ int i;
+
+ /*
+ * Try to locate a physical channel to be used for
+ * this transfer. If all are taken return NULL and
+ * the requester will have to cope by using some fallback
+ * PIO mode or retrying later.
+ */
+ for (i = 0; i < pl08x->vd->channels; i++) {
+ ch = &pl08x->phy_chans[i];
+
+ spin_lock_irqsave(&ch->lock, flags);
+
+ if (!ch->serving) {
+ ch->serving = virt_chan;
+ ch->signal = -1;
+ spin_unlock_irqrestore(&ch->lock, flags);
+ break;
+ }
+
+ spin_unlock_irqrestore(&ch->lock, flags);
+ }
+
+ if (i == pl08x->vd->channels) {
+ /* No physical channel available, cope with it */
+ return NULL;
+ }
+
+ return ch;
+}
+
+static inline void pl08x_put_phy_channel(struct pl08x_driver_data *pl08x,
+ struct pl08x_phy_chan *ch)
+{
+ unsigned long flags;
+
+ /* Stop the channel and clear its interrupts */
+ pl08x_stop_phy_chan(ch);
+ writel((1 << ch->id), pl08x->base + PL080_ERR_CLEAR);
+ writel((1 << ch->id), pl08x->base + PL080_TC_CLEAR);
+
+ /* Mark it as free */
+ spin_lock_irqsave(&ch->lock, flags);
+ ch->serving = NULL;
+ ch->signal = -1;
+ spin_unlock_irqrestore(&ch->lock, flags);
+}
+
+/*
+ * LLI handling
+ */
+
+static inline unsigned int pl08x_get_bytes_for_cctl(unsigned int coded)
+{
+ switch (coded) {
+ case PL080_WIDTH_8BIT:
+ return 1;
+ case PL080_WIDTH_16BIT:
+ return 2;
+ case PL080_WIDTH_32BIT:
+ return 4;
+ default:
+ break;
+ }
+ BUG();
+ return 0;
+}
+
+static inline u32 pl08x_cctl_bits(u32 cctl,
+ u8 srcwidth,
+ u8 dstwidth,
+ u32 tsize)
+{
+ u32 retbits = cctl;
+
+ /* Remove all src, dst and transfersize bits */
+ retbits &= ~PL080_CONTROL_DWIDTH_MASK;
+ retbits &= ~PL080_CONTROL_SWIDTH_MASK;
+ retbits &= ~PL080_CONTROL_TRANSFER_SIZE_MASK;
+
+ /* Then set the bits according to the parameters */
+ switch(srcwidth) {
+ case 1:
+ retbits |= PL080_WIDTH_8BIT << PL080_CONTROL_SWIDTH_SHIFT;
+ break;
+ case 2:
+ retbits |= PL080_WIDTH_16BIT << PL080_CONTROL_SWIDTH_SHIFT;
+ break;
+ case 4:
+ retbits |= PL080_WIDTH_32BIT << PL080_CONTROL_SWIDTH_SHIFT;
+ break;
+ default:
+ BUG();
+ break;
+ }
+
+ switch(dstwidth) {
+ case 1:
+ retbits |= PL080_WIDTH_8BIT << PL080_CONTROL_DWIDTH_SHIFT;
+ break;
+ case 2:
+ retbits |= PL080_WIDTH_16BIT << PL080_CONTROL_DWIDTH_SHIFT;
+ break;
+ case 4:
+ retbits |= PL080_WIDTH_32BIT << PL080_CONTROL_DWIDTH_SHIFT;
+ break;
+ default:
+ BUG();
+ break;
+ }
+
+ retbits |= tsize << PL080_CONTROL_TRANSFER_SIZE_SHIFT;
+ return retbits;
+}
+
+/*
+ * Autoselect a master bus to use for the transfer
+ * this prefers the destination bus if both available
+ * if fixed address on one bus the other will be chosen
+ */
+void pl08x_choose_master_bus(struct pl08x_bus_data *src_bus,
+ struct pl08x_bus_data *dst_bus, struct pl08x_bus_data **mbus,
+ struct pl08x_bus_data **sbus, u32 cctl)
+{
+ if (!cctl & PL080_CONTROL_DST_INCR) {
+ *mbus = src_bus;
+ *sbus = dst_bus;
+ } else if (!cctl & PL080_CONTROL_SRC_INCR) {
+ *mbus = dst_bus;
+ *sbus = src_bus;
+ } else {
+ if (dst_bus->buswidth == 4) {
+ *mbus = dst_bus;
+ *sbus = src_bus;
+ } else if (src_bus->buswidth == 4) {
+ *mbus = src_bus;
+ *sbus = dst_bus;
+ } else if (dst_bus->buswidth == 2) {
+ *mbus = dst_bus;
+ *sbus = src_bus;
+ } else if (src_bus->buswidth == 2) {
+ *mbus = src_bus;
+ *sbus = dst_bus;
+ } else {
+ /* src_bus->buswidth == 1 */
+ *mbus = dst_bus;
+ *sbus = src_bus;
+ }
+ }
+}
+
+/*
+ * Fills in one LLI for a certain transfer descriptor
+ * and advance the counter
+ */
+int pl08x_fill_lli_for_desc(struct pl08x_driver_data *pl08x,
+ struct pl08x_txd *txd, int num_llis, int len,
+ u32 cctl, u32 *remainder)
+{
+ struct lli *llis_va = (struct lli *)(txd->llis_va);
+ struct lli *llis_bus = (struct lli *)(txd->llis_bus);
+
+ BUG_ON(num_llis >= MAX_NUM_TSFR_LLIS);
+
+ llis_va[num_llis].cctl = cctl;
+ llis_va[num_llis].src = txd->srcbus.addr;
+ llis_va[num_llis].dst = txd->dstbus.addr;
+ /*
+ * The bus bit is added to the next lli's address
+ */
+ llis_va[num_llis].next =
+ (dma_addr_t)((u32) &(llis_bus[num_llis + 1])
+ | pl08x->pd->bus_bit_lli);
+
+ if (cctl & PL080_CONTROL_SRC_INCR)
+ txd->srcbus.addr += len;
+ if (cctl & PL080_CONTROL_DST_INCR)
+ txd->dstbus.addr += len;
+
+ *remainder -= len;
+
+ return num_llis + 1;
+}
+
+/*
+ * Return number of bytes to fill to boundary, or len
+ */
+static inline u32 pl08x_pre_boundary(u32 addr, u32 len)
+{
+ u32 boundary;
+
+ boundary = ((addr >> PL08X_BOUNDARY_SHIFT) + 1)
+ << PL08X_BOUNDARY_SHIFT;
+
+ if (boundary < addr + len)
+ return boundary - addr;
+ else
+ return len;
+}
+
+/*
+ * This fills in the table of LLIs for the transfer descriptor
+ * Note that we assume we never have to change the burst sizes
+ * Return 0 for error
+ */
+static int pl08x_fill_llis_for_desc(struct pl08x_driver_data *pl08x,
+ struct pl08x_txd *txd)
+{
+ struct pl08x_channel_data *cd = txd->cd;
+ struct pl08x_bus_data *mbus, *sbus;
+ u32 remainder;
+ int num_llis = 0;
+ u32 cctl;
+ int max_bytes_per_lli;
+ int total_bytes = 0;
+ struct lli *llis_va;
+ struct lli *llis_bus;
+
+ if (!txd) {
+ dev_err(&pl08x->adev->dev, "%s no descriptor\n", __func__);
+ return 0;
+ }
+
+ txd->llis_va = dma_pool_alloc(pl08x->pool, GFP_KERNEL,
+ &txd->llis_bus);
+ if (!txd->llis_va) {
+ dev_err(&pl08x->adev->dev, "%s no memory for llis\n", __func__);
+ return 0;
+ }
+
+ pl08x->pool_ctr++;
+
+ /*
+ * Initialize bus values for this transfer
+ * from the passed optimal values
+ */
+ if (!cd) {
+ dev_err(&pl08x->adev->dev, "%s no channel data\n", __func__);
+ return 0;
+ }
+
+ /* Get the default CCTL from the platform data */
+ cctl = cd->cctl;
+
+ /* Find maximum width of the source bus */
+ txd->srcbus.maxwidth =
+ pl08x_get_bytes_for_cctl((cctl & PL080_CONTROL_SWIDTH_MASK) >>
+ PL080_CONTROL_SWIDTH_SHIFT);
+
+ /* Find maximum width of the destination bus */
+ txd->dstbus.maxwidth =
+ pl08x_get_bytes_for_cctl((cctl & PL080_CONTROL_DWIDTH_MASK) >>
+ PL080_CONTROL_DWIDTH_SHIFT);
+
+ /* Set up the bus widths to the maximum */
+ txd->srcbus.buswidth = txd->srcbus.maxwidth;
+ txd->dstbus.buswidth = txd->dstbus.maxwidth;
+ dev_vdbg(&pl08x->adev->dev,
+ "%s source bus is %d bytes wide, dest bus is %d bytes wide\n",
+ __func__, txd->srcbus.buswidth, txd->dstbus.buswidth);
+
+
+ /*
+ * bytes transferred == tsize * MIN(buswidths), not max(buswidths)
+ */
+ max_bytes_per_lli = min(txd->srcbus.buswidth, txd->dstbus.buswidth) *
+ PL080_CONTROL_TRANSFER_SIZE_MASK;
+ dev_vdbg(&pl08x->adev->dev,
+ "%s max bytes per lli = %d\n",
+ __func__, max_bytes_per_lli);
+
+ /* We need to count this down to zero */
+ remainder = txd->len;
+ dev_vdbg(&pl08x->adev->dev,
+ "%s remainder = %d\n",
+ __func__, remainder);
+
+ /*
+ * Choose bus to align to
+ * - prefers destination bus if both available
+ * - if fixed address on one bus chooses other
+ */
+ pl08x_choose_master_bus(&txd->srcbus,
+ &txd->dstbus, &mbus, &sbus, cctl);
+
+ if (txd->len < mbus->buswidth) {
+ /*
+ * Less than a bus width available
+ * - send as single bytes
+ */
+ while (remainder) {
+ dev_vdbg(&pl08x->adev->dev,
+ "%s single byte LLIs for a transfer of less than a bus width (remain %08x)\n",
+ __func__, remainder);
+ cctl = pl08x_cctl_bits(cctl, 1, 1, 1);
+ num_llis =
+ pl08x_fill_lli_for_desc(pl08x, txd, num_llis, 1,
+ cctl, &remainder);
+ total_bytes++;
+ }
+ } else {
+ /*
+ * Make one byte LLIs until master bus is aligned
+ * - slave will then be aligned also
+ */
+ while ((mbus->addr) % (mbus->buswidth)) {
+ dev_vdbg(&pl08x->adev->dev,
+ "%s adjustment lli for less than bus width (remain %08x)\n",
+ __func__, remainder);
+ cctl = pl08x_cctl_bits(cctl, 1, 1, 1);
+ num_llis = pl08x_fill_lli_for_desc
+ (pl08x, txd, num_llis, 1, cctl, &remainder);
+ total_bytes++;
+ }
+
+ /*
+ * Master now aligned
+ * - if slave is not then we must set its width down
+ */
+ if (sbus->addr % sbus->buswidth) {
+ dev_dbg(&pl08x->adev->dev,
+ "%s set down bus width to one byte\n",
+ __func__);
+
+ sbus->buswidth = 1;
+ }
+
+ /*
+ * Make largest possible LLIs until less than one bus width left
+ */
+ while (remainder > (mbus->buswidth - 1)) {
+ int lli_len, target_len;
+ int tsize;
+ int odd_bytes;
+
+ /*
+ * If enough left try to send max possible,
+ * otherwise try to send the remainder
+ */
+ target_len = remainder;
+ if (remainder > max_bytes_per_lli)
+ target_len = max_bytes_per_lli;
+
+ /*
+ * Set bus lengths for incrementing busses
+ * to number of bytes which fill to next memory
+ * boundary
+ */
+ if (cctl & PL080_CONTROL_SRC_INCR)
+ txd->srcbus.fill_bytes =
+ pl08x_pre_boundary(
+ txd->srcbus.addr,
+ remainder);
+ else
+ txd->srcbus.fill_bytes =
+ max_bytes_per_lli;
+
+ if (cctl & PL080_CONTROL_DST_INCR)
+ txd->dstbus.fill_bytes =
+ pl08x_pre_boundary(
+ txd->dstbus.addr,
+ remainder);
+ else
+ txd->dstbus.fill_bytes =
+ max_bytes_per_lli;
+
+ /*
+ * Find the nearest
+ */
+ lli_len = min(txd->srcbus.fill_bytes,
+ txd->dstbus.fill_bytes);
+
+ BUG_ON(lli_len > remainder);
+
+ if (lli_len <= 0) {
+ dev_err(&pl08x->adev->dev,
+ "%s lli_len is %d, <= 0\n",
+ __func__, lli_len);
+ return 0;
+ }
+
+ if (lli_len == target_len) {
+ /*
+ * Can send what we wanted
+ */
+ /*
+ * Maintain alignment
+ */
+ lli_len = (lli_len/mbus->buswidth) *
+ mbus->buswidth;
+ odd_bytes = 0;
+ } else {
+ /*
+ * So now we know how many bytes to transfer
+ * to get to the nearest boundary
+ * The next lli will past the boundary
+ * - however we may be working to a boundary
+ * on the slave bus
+ * We need to ensure the master stays aligned
+ */
+ odd_bytes = lli_len % mbus->buswidth;
+ /*
+ * - and that we are working in multiples
+ * of the bus widths
+ */
+ lli_len -= odd_bytes;
+
+ }
+
+ if (lli_len) {
+ /*
+ * Check against minimum bus alignment:
+ * Calculate actual transfer size in relation to bus
+ * width an get a maximum remainder of the smallest
+ * bus width - 1
+ */
+ /* FIXME: use round_down()? */
+ tsize = lli_len / min(mbus->buswidth, sbus->buswidth);
+ lli_len = tsize * min(mbus->buswidth, sbus->buswidth);
+
+ if (target_len != lli_len) {
+ dev_vdbg(&pl08x->adev->dev,
+ "%s can't send what we want. Desired %08x, lli of %08x bytes in txd of %08x\n",
+ __func__, target_len, lli_len, txd->len);
+ }
+
+ cctl = pl08x_cctl_bits(cctl,
+ txd->srcbus.buswidth,
+ txd->dstbus.buswidth,
+ tsize);
+
+ dev_vdbg(&pl08x->adev->dev,
+ "%s fill lli with single lli chunk of size %08x (remainder %08x)\n",
+ __func__, lli_len, remainder);
+ num_llis = pl08x_fill_lli_for_desc(pl08x, txd,
+ num_llis, lli_len, cctl,
+ &remainder);
+ total_bytes += lli_len;
+ }
+
+
+ if (odd_bytes) {
+ /*
+ * Creep past the boundary,
+ * maintaining master alignment
+ */
+ int j;
+ for (j = 0; (j < mbus->buswidth)
+ && (remainder); j++) {
+ cctl = pl08x_cctl_bits(cctl, 1, 1, 1);
+ dev_vdbg(&pl08x->adev->dev,
+ "%s align with boundardy, single byte (remain %08x)\n",
+ __func__, remainder);
+ num_llis =
+ pl08x_fill_lli_for_desc(pl08x,
+ txd, num_llis, 1,
+ cctl, &remainder);
+ total_bytes++;
+ }
+ }
+ }
+
+ /*
+ * Send any odd bytes
+ */
+ if (remainder < 0) {
+ dev_err(&pl08x->adev->dev, "%s remainder not fitted 0x%08x bytes\n",
+ __func__, remainder);
+ return 0;
+ }
+
+ while (remainder) {
+ cctl = pl08x_cctl_bits(cctl, 1, 1, 1);
+ dev_vdbg(&pl08x->adev->dev,
+ "%s align with boundardy, single odd byte (remain %d)\n",
+ __func__, remainder);
+ num_llis = pl08x_fill_lli_for_desc(pl08x, txd, num_llis,
+ 1, cctl, &remainder);
+ total_bytes++;
+ }
+ }
+ if (total_bytes != txd->len) {
+ dev_err(&pl08x->adev->dev,
+ "%s size of encoded lli:s don't match total txd, transferred 0x%08x from size 0x%08x\n",
+ __func__, total_bytes, txd->len);
+ return 0;
+ }
+
+ if (num_llis >= MAX_NUM_TSFR_LLIS) {
+ dev_err(&pl08x->adev->dev,
+ "%s need to increase MAX_NUM_TSFR_LLIS from 0x%08x\n",
+ __func__, (u32) MAX_NUM_TSFR_LLIS);
+ return 0;
+ }
+ /*
+ * Decide whether this is a loop or a terminated transfer
+ */
+ llis_va = ((struct lli *)txd->llis_va);
+ llis_bus = ((struct lli *)txd->llis_bus);
+
+ if (cd->circular_buffer) {
+ /*
+ * Loop the circular buffer so that the next element
+ * points back to the beginning of the LLI.
+ */
+ llis_va[num_llis - 1].next =
+ (dma_addr_t)((unsigned int)&(llis_bus[0]) +
+ pl08x->pd->bus_bit_lli);
+ } else {
+ /*
+ * On non-circular buffers, the final LLI terminates
+ * the LLI.
+ */
+ llis_va[num_llis - 1].next = 0;
+ /*
+ * The final LLI element shall also fire an interrupt
+ */
+ llis_va[num_llis - 1].cctl |= PL080_CONTROL_TC_IRQ_EN;
+ }
+
+ /* Now store the channel register values */
+ txd->csrc = llis_va[0].src;
+ txd->cdst = llis_va[0].dst;
+ if (num_llis > 1)
+ txd->clli = llis_va[0].next;
+ else
+ txd->clli = 0;
+
+ txd->cctl = llis_va[0].cctl;
+ /* ccfg will be set at physical channel allocation time */
+
+ {
+ int i;
+
+ for (i = 0; i < num_llis; i++) {
+ dev_vdbg(&pl08x->adev->dev,
+ "lli %d @%p: csrc=%08x, cdst=%08x, cctl=%08x, clli=%08x\n",
+ i,
+ &llis_va[i],
+ llis_va[i].src,
+ llis_va[i].dst,
+ llis_va[i].cctl,
+ llis_va[i].next
+ );
+ }
+ }
+
+ /*
+ * Reflects the longest lli submitted so far
+ * TODO: Change to use /proc data
+ */
+ if (pl08x->max_num_llis < num_llis)
+ pl08x->max_num_llis = num_llis;
+
+ return num_llis;
+}
+
+/* You should call this with the struct pl08x lock held */
+static void pl08x_free_txd(struct pl08x_driver_data *pl08x, struct pl08x_txd *txd)
+{
+ if (!txd)
+ dev_err(&pl08x->adev->dev,
+ "%s no descriptor to free\n",
+ __func__);
+
+ /* Free the LLI */
+ dma_pool_free(pl08x->pool, txd->llis_va,
+ txd->llis_bus);
+
+ pl08x->pool_ctr--;
+
+ kfree(txd);
+}
+
+static void pl08x_free_txd_list(struct pl08x_driver_data *pl08x, struct list_head *txdlist)
+{
+ struct pl08x_txd *txdi = NULL;
+ struct pl08x_txd *next;
+
+ if (!list_empty(txdlist)) {
+ list_for_each_entry_safe(txdi,
+ next, txdlist, node) {
+ list_del(&txdi->node);
+ pl08x_free_txd(pl08x, txdi);
+ }
+
+ }
+}
+
+static void pl08x_tasklet(unsigned long data)
+{
+ struct pl08x_dma_chan *plchan = (struct pl08x_dma_chan *) data;
+ struct pl08x_phy_chan *phychan = plchan->phychan;
+ struct pl08x_driver_data *pl08x = plchan->host;
+ struct pl08x_txd *txdi = NULL;
+ struct pl08x_txd *next;
+ unsigned long flags;
+
+ if (!plchan)
+ BUG();
+
+ spin_lock_irqsave(&plchan->lock, flags);
+
+ if (plchan->at) {
+ dma_async_tx_callback callback =
+ plchan->at->tx.callback;
+ void *callback_param =
+ plchan->at->tx.callback_param;
+
+ /*
+ * Update last completed
+ */
+ plchan->lc =
+ (plchan->at->tx.cookie);
+
+ /*
+ * Callback peripheral driver for p/m
+ * to signal completion
+ */
+ if (callback)
+ callback(callback_param);
+
+ /*
+ * Device callbacks should NOT clear
+ * the current transaction on the channel
+ * Linus: sometimes they should?
+ */
+ if (!plchan->at)
+ BUG();
+
+ /*
+ * Free the descriptor if it's not for a device
+ * using a circular buffer
+ */
+ if (!plchan->at->cd->circular_buffer) {
+ pl08x_free_txd(pl08x, plchan->at);
+ plchan->at = NULL;
+ }
+ /*
+ * else descriptor for circular
+ * buffers only freed when
+ * client has disabled dma
+ */
+ }
+ /*
+ * If a new descriptor is queued, set it up
+ */
+ if (!list_empty(&plchan->desc_list)) {
+ list_for_each_entry_safe(txdi,
+ next, &plchan->desc_list, node) {
+ list_del_init(&txdi->node);
+ }
+ } else {
+ /*
+ * No more jobs, so free up the physical channel
+ * Free any allocated signal on slave transfers too
+ */
+ if ((phychan->signal >= 0) && pl08x->pd->put_signal)
+ pl08x->pd->put_signal(plchan);
+ pl08x_put_phy_channel(pl08x, phychan);
+ plchan->phychan = NULL;
+ }
+
+ spin_unlock_irqrestore(&plchan->lock, flags);
+}
+
+static irqreturn_t pl08x_irq(int irq, void *dev)
+{
+ struct pl08x_driver_data *pl08x = dev;
+ u32 mask = 0;
+ u32 val;
+ int i;
+
+ val = readl(pl08x->base + PL080_ERR_STATUS);
+ mb();
+ if (val) {
+ /*
+ * An error interrupt (on one or more channels)
+ */
+ dev_err(&pl08x->adev->dev,
+ "%s error interrupt, register value 0x%08x\n",
+ __func__, val);
+ /*
+ * Simply clear ALL PL08X error interrupts,
+ * regardless of channel and cause
+ * FIXME: should be 0x00000003 on PL081 really.
+ */
+ writel(0x000000FF, pl08x->base + PL080_ERR_CLEAR);
+ }
+ val = readl(pl08x->base + PL080_INT_STATUS);
+ mb();
+ for (i = 0; i < pl08x->vd->channels; i++) {
+ if ((1 << i) & val) {
+ /* Locate physical channel */
+ struct pl08x_phy_chan *phychan = &pl08x->phy_chans[i];
+ struct pl08x_dma_chan *plchan = phychan->serving;
+
+ /* Schedule tasklet on this channel */
+ tasklet_schedule(&plchan->tasklet);
+
+ mask |= (1 << i);
+ }
+ }
+ /*
+ * Clear only the terminal interrupts on channels we processed
+ */
+ writel(mask, pl08x->base + PL080_TC_CLEAR);
+ mb();
+
+ return mask ? IRQ_HANDLED : IRQ_NONE;
+}
+
+
+/*
+ * The DMA ENGINE API
+ */
+static int pl08x_alloc_chan_resources(struct dma_chan *chan)
+{
+ return 0;
+}
+
+static void pl08x_free_chan_resources(struct dma_chan *chan)
+{
+}
+
+/*
+ * This should be called with the channel plchan->lock held
+ */
+static int prep_phy_channel(struct pl08x_dma_chan *plchan,
+ struct pl08x_txd *txd)
+{
+ struct pl08x_driver_data *pl08x = plchan->host;
+ struct pl08x_phy_chan *ch;
+ int ret;
+
+ /* Check if we already have a channel */
+ if (plchan->phychan)
+ return 0;
+
+ ch = pl08x_get_phy_channel(pl08x, plchan);
+ if (!ch) {
+ /* No physical channel available, cope with it */
+ dev_info(&pl08x->adev->dev, "no physical channel "
+ "available for xfer on %s\n", plchan->name);
+ return -EBUSY;
+ }
+
+ /*
+ * OK we have a physical channel: for memcpy() this is all we
+ * need, but for slaves the physical siglals may be muxed!
+ * Can the platform allow us to use this channel?
+ */
+ if ((txd->direction == DMA_FROM_DEVICE || txd->direction == DMA_TO_DEVICE) &&
+ pl08x->pd->get_signal) {
+ ret = pl08x->pd->get_signal(plchan);
+ if (ret < 0) {
+ dev_info(&pl08x->adev->dev,
+ "unable to use physical channel "
+ "%d for transfer on %s due to "
+ "platform restrictions\n",
+ ch->id, plchan->name);
+ /* Release physical channel & return */
+ pl08x_put_phy_channel(pl08x, ch);
+ return -EBUSY;
+ }
+ ch->signal = ret;
+ }
+
+ dev_dbg(&pl08x->adev->dev, "allocated physical "
+ "channel %d and signal %d for xfer on %s\n",
+ ch->id,
+ ch->signal,
+ plchan->name);
+
+ plchan->phychan = ch;
+
+ return 0;
+}
+
+/*
+ * First make the LLIs (could/should we do this earlier??)
+ * slave (m/p) - no queued transactions allowed at present
+ * TODO allow queued transactions for non circular buffers
+ * Set up the channel active txd as inactive
+ * m2m - transactions may be queued
+ * If no active txd on channel
+ * set it up as inactive
+ * - issue_pending() will set active & start
+ * else
+ * queue it
+ * Lock channel since there may be (at least for m2m) multiple calls
+ *
+ * Return < 0 for error
+ */
+
+static dma_cookie_t pl08x_tx_submit(struct dma_async_tx_descriptor *tx)
+{
+ int num_llis;
+ unsigned long flags;
+ struct pl08x_txd *txd = container_of(tx, struct pl08x_txd, tx);
+ struct pl08x_dma_chan *plchan = to_pl08x_chan(tx->chan);
+ struct pl08x_driver_data *pl08x = plchan->host;
+ int ret;
+
+ num_llis = pl08x_fill_llis_for_desc(pl08x, txd);
+
+ if (num_llis) {
+ spin_lock_irqsave(&plchan->lock, flags);
+ atomic_inc(&plchan->last_issued);
+ tx->cookie = atomic_read(&plchan->last_issued);
+
+ if (plchan->at) {
+
+ /*
+ * If this device not using a circular buffer then
+ * queue this new descriptor for transfer.
+ * The descriptor for a circular buffer continues
+ * to be used until the channel is freed.
+ */
+ if (txd->cd->circular_buffer)
+ dev_err(&pl08x->adev->dev,
+ "%s attempting to queue a circular buffer\n",
+ __func__);
+ else
+ list_add_tail(&txd->node,
+ &plchan->desc_list);
+
+ } else {
+ plchan->at = txd;
+ txd->active = false;
+ }
+
+ /*
+ * See if we already have a physical channel allocated,
+ * else this is the time to try to get one.
+ */
+ ret = prep_phy_channel(plchan, txd);
+ if (ret) {
+ /* No physical channel available, cope with it */
+ spin_unlock_irqrestore(&plchan->lock, flags);
+ return -EBUSY;
+ }
+
+ spin_unlock_irqrestore(&plchan->lock, flags);
+
+ return tx->cookie;
+ } else
+ return -EINVAL;
+}
+
+static struct dma_async_tx_descriptor *pl08x_prep_dma_interrupt(
+ struct dma_chan *chan, unsigned long flags)
+{
+ struct dma_async_tx_descriptor *retval = NULL;
+
+ return retval;
+}
+
+/*
+ * Code accessing dma_async_is_complete() in a tight loop
+ * may give problems - could schedule where indicated.
+ * If slaves are relying on interrupts to signal completion this
+ * function must not be called with interrupts disabled
+ */
+static enum dma_status
+pl08x_dma_tx_status(struct dma_chan *chan,
+ dma_cookie_t cookie,
+ struct dma_tx_state *txstate)
+{
+ struct pl08x_dma_chan *plchan = to_pl08x_chan(chan);
+ dma_cookie_t last_used;
+ dma_cookie_t last_complete;
+ enum dma_status ret;
+ u32 bytesleft = 0;
+
+ last_used = atomic_read(&plchan->last_issued);
+ last_complete = plchan->lc;
+
+ ret = dma_async_is_complete(cookie, last_complete, last_used);
+ if (ret == DMA_SUCCESS) {
+ dma_set_tx_state(txstate, last_complete, last_used, 0);
+ return ret;
+ }
+
+ /*
+ * schedule(); could be inserted here
+ */
+
+ /*
+ * This cookie not complete yet
+ */
+ last_used = atomic_read(&plchan->last_issued);
+ last_complete = plchan->lc;
+
+ /* Get number of bytes left in the active transaction */
+ if (plchan->phychan)
+ bytesleft = pl08x_getbytes_phy_chan(plchan->phychan);
+
+ dma_set_tx_state(txstate, last_complete, last_used,
+ bytesleft);
+
+ return DMA_IN_PROGRESS;
+ /* FIXME: make possible to return DMA_IN_PROGRESS */
+}
+
+/* PrimeCell DMA extension */
+struct burst_table {
+ int burstwords;
+ u32 reg;
+};
+
+static const struct burst_table burst_sizes[] = {
+ {
+ .burstwords = 256,
+ .reg = (PL080_BSIZE_256 << PL080_CONTROL_SB_SIZE_SHIFT) |
+ (PL080_BSIZE_256 << PL080_CONTROL_DB_SIZE_SHIFT),
+ },
+ {
+ .burstwords = 128,
+ .reg = (PL080_BSIZE_128 << PL080_CONTROL_SB_SIZE_SHIFT) |
+ (PL080_BSIZE_128 << PL080_CONTROL_DB_SIZE_SHIFT),
+ },
+ {
+ .burstwords = 64,
+ .reg = (PL080_BSIZE_64 << PL080_CONTROL_SB_SIZE_SHIFT) |
+ (PL080_BSIZE_64 << PL080_CONTROL_DB_SIZE_SHIFT),
+ },
+ {
+ .burstwords = 32,
+ .reg = (PL080_BSIZE_32 << PL080_CONTROL_SB_SIZE_SHIFT) |
+ (PL080_BSIZE_32 << PL080_CONTROL_DB_SIZE_SHIFT),
+ },
+ {
+ .burstwords = 16,
+ .reg = (PL080_BSIZE_16 << PL080_CONTROL_SB_SIZE_SHIFT) |
+ (PL080_BSIZE_16 << PL080_CONTROL_DB_SIZE_SHIFT),
+ },
+ {
+ .burstwords = 8,
+ .reg = (PL080_BSIZE_8 << PL080_CONTROL_SB_SIZE_SHIFT) |
+ (PL080_BSIZE_8 << PL080_CONTROL_DB_SIZE_SHIFT),
+ },
+ {
+ .burstwords = 4,
+ .reg = (PL080_BSIZE_4 << PL080_CONTROL_SB_SIZE_SHIFT) |
+ (PL080_BSIZE_4 << PL080_CONTROL_DB_SIZE_SHIFT),
+ },
+ {
+ .burstwords = 1,
+ .reg = (PL080_BSIZE_1 << PL080_CONTROL_SB_SIZE_SHIFT) |
+ (PL080_BSIZE_1 << PL080_CONTROL_DB_SIZE_SHIFT),
+ },
+};
+
+static void dma_set_ambaconfig(struct dma_chan *chan,
+ struct amba_dma_channel_config *config)
+{
+ struct pl08x_dma_chan *plchan = to_pl08x_chan(chan);
+ struct pl08x_driver_data *pl08x = plchan->host;
+ struct pl08x_channel_data *cd = plchan->cd;
+ int maxburst = config->maxburst;
+ u32 cctl = 0;
+ /* Mask out all except src and dst channel */
+ u32 ccfg = cd->ccfg & 0x000003DEU;
+ int i = 0;
+
+ plchan->amba_addr = config->addr;
+ plchan->amba_direction = config->direction;
+
+ switch (config->addr_width) {
+ case 1:
+ cctl |= (PL080_WIDTH_8BIT << PL080_CONTROL_SWIDTH_SHIFT) |
+ (PL080_WIDTH_8BIT << PL080_CONTROL_DWIDTH_SHIFT);
+ break;
+ case 2:
+ cctl |= (PL080_WIDTH_16BIT << PL080_CONTROL_SWIDTH_SHIFT) |
+ (PL080_WIDTH_16BIT << PL080_CONTROL_DWIDTH_SHIFT);
+ break;
+ case 4:
+ cctl |= (PL080_WIDTH_32BIT << PL080_CONTROL_SWIDTH_SHIFT) |
+ (PL080_WIDTH_32BIT << PL080_CONTROL_DWIDTH_SHIFT);
+ break;
+ default:
+ dev_err(&pl08x->adev->dev,
+ "bad ambaconfig: alien address width\n");
+ return;
+ }
+
+ /* Now decide on a maxburst */
+ while (i < ARRAY_SIZE(burst_sizes)) {
+ if (burst_sizes[i].burstwords <= maxburst)
+ break;
+ i++;
+ }
+ cctl |= burst_sizes[i].reg;
+
+ /* Transfer direction */
+ if (config->direction == DMA_TO_DEVICE) {
+ cctl |= PL080_CONTROL_SRC_INCR;
+ ccfg |= PL080_FLOW_MEM2PER << PL080_CONFIG_FLOW_CONTROL_SHIFT;
+ } else if (config->direction == DMA_FROM_DEVICE) {
+ cctl |= PL080_CONTROL_DST_INCR;
+ ccfg |= PL080_FLOW_PER2MEM << PL080_CONFIG_FLOW_CONTROL_SHIFT;
+ } else {
+ dev_err(&pl08x->adev->dev,
+ "bad ambaconfig: alien transfer direction\n");
+ }
+
+ /* Access the cell in privileged mode, non-bufferable, non-cacheable */
+ cctl &= ~PL080_CONTROL_PROT_MASK;
+ cctl |= PL080_CONTROL_PROT_SYS;
+
+ /* Modify the default channel data to fit PrimeCell request */
+ cd->cctl = cctl;
+ cd->ccfg = ccfg;
+
+ dev_info(&pl08x->adev->dev,
+ "configured channel %s (%s) for %s, data width %d, "
+ "maxburst %d words, LE, CCTL=%08x, CCFG=%08x\n",
+ dma_chan_name(chan), plchan->name,
+ (config->direction == DMA_FROM_DEVICE) ? "RX" : "TX",
+ config->addr_width,
+ config->maxburst,
+ cctl, ccfg);
+}
+
+/*
+ * Slave transactions callback to the slave device to allow
+ * synchronization of slave DMA signals with the DMAC enable
+ */
+static void pl08x_issue_pending(struct dma_chan *chan)
+{
+ struct pl08x_dma_chan *plchan = to_pl08x_chan(chan);
+ struct pl08x_driver_data *pl08x = plchan->host;
+ unsigned long flags;
+
+ spin_lock_irqsave(&plchan->lock, flags);
+ if (plchan->at) {
+ if (!plchan->at->active) {
+ /* Configure the physical channel for the active txd */
+ pl08x_config_phychan_for_txd(plchan);
+ pl08x_set_cregs(pl08x, plchan->phychan);
+ pl08x_enable_phy_chan(pl08x, plchan->phychan);
+ plchan->at->active = true;
+ }
+ /*
+ * else skip active transfer
+ * Calls with active txd occur for NET_DMA
+ * - there can be queued descriptors
+ */
+ }
+ spin_unlock_irqrestore(&plchan->lock, flags);
+ /*
+ * else - calls with no active descriptor occur for NET_DMA
+ */
+}
+
+/*
+ * Initialize a descriptor to be used by memcpy submit
+ */
+static struct dma_async_tx_descriptor *pl08x_prep_dma_memcpy(
+ struct dma_chan *chan, dma_addr_t dest, dma_addr_t src,
+ size_t len, unsigned long flags)
+{
+ struct pl08x_dma_chan *plchan = to_pl08x_chan(chan);
+ struct pl08x_driver_data *pl08x = plchan->host;
+ struct pl08x_txd *txd;
+
+ txd = kzalloc(sizeof(struct pl08x_txd), GFP_KERNEL);
+ if (!txd) {
+ dev_err(&pl08x->adev->dev,
+ "%s no memory for descriptor\n", __func__);
+ return NULL;
+ }
+
+ dma_async_tx_descriptor_init(&txd->tx, chan);
+ txd->direction = DMA_NONE;
+ txd->srcbus.addr = src;
+ txd->dstbus.addr = dest;
+
+ /* Set platform data for m2m */
+ txd->cd = &pl08x->pd->memcpy_channel;
+ /* Both to be incremented or the code will break */
+ txd->cd->cctl |= PL080_CONTROL_SRC_INCR | PL080_CONTROL_DST_INCR;
+ txd->tx.tx_submit = pl08x_tx_submit;
+ txd->tx.callback = NULL;
+ txd->tx.callback_param = NULL;
+ txd->len = len;
+
+ INIT_LIST_HEAD(&txd->node);
+
+ return &txd->tx;
+}
+
+struct dma_async_tx_descriptor *pl08x_prep_slave_sg(
+ struct dma_chan *chan, struct scatterlist *sgl,
+ unsigned int sg_len, enum dma_data_direction direction,
+ unsigned long flags)
+{
+ struct pl08x_dma_chan *plchan = to_pl08x_chan(chan);
+ struct pl08x_driver_data *pl08x = plchan->host;
+ struct pl08x_txd *txd;
+
+ /*
+ * Current implementation ASSUMES only one sg
+ */
+ if (sg_len != 1) {
+ dev_err(&pl08x->adev->dev, "%s prepared too long sglist\n",
+ __func__);
+ BUG();
+ }
+
+ dev_info(&pl08x->adev->dev, "%s prepare transaction from %s\n",
+ __func__, plchan->name);
+
+ txd = kmalloc(sizeof(struct pl08x_txd), GFP_KERNEL);
+ if (!txd) {
+ dev_err(&pl08x->adev->dev, "%s no txd\n", __func__);
+ return NULL;
+ }
+
+ dma_async_tx_descriptor_init(&txd->tx, chan);
+
+ if (direction != plchan->amba_direction)
+ dev_err(&pl08x->adev->dev, "%s DMA setup does not match "
+ "the direction configured for the PrimeCell\n",
+ __func__);
+
+ txd->direction = direction;
+ if (direction == DMA_TO_DEVICE) {
+ txd->srcbus.addr = sgl->dma_address;
+ txd->dstbus.addr = plchan->amba_addr;
+ } else if (direction == DMA_FROM_DEVICE) {
+ txd->srcbus.addr = plchan->amba_addr;
+ txd->dstbus.addr = sgl->dma_address;
+ } else {
+ dev_err(&pl08x->adev->dev,
+ "%s direction unsupported\n", __func__);
+ return NULL;
+ }
+ txd->cd = plchan->cd;
+ txd->tx.tx_submit = pl08x_tx_submit;
+ txd->tx.callback = NULL;
+ txd->tx.callback_param = NULL;
+ txd->len = sgl->length;
+
+ INIT_LIST_HEAD(&txd->node);
+
+ return &txd->tx;
+}
+
+static int pl08x_control(struct dma_chan *chan, enum dma_ctrl_cmd cmd,
+ unsigned long arg)
+{
+ struct pl08x_dma_chan *plchan = to_pl08x_chan(chan);
+ struct pl08x_driver_data *pl08x = plchan->host;
+ unsigned long flags;
+
+ /* Controls applicable to inactive channels */
+ if (cmd == DMA_CONFIG_AMBA) {
+ dma_set_ambaconfig(chan,
+ (struct amba_dma_channel_config *)
+ arg);
+ return 0;
+ }
+
+ /* Anything succeeds on non-existing transfers */
+ spin_lock_irqsave(&plchan->lock, flags);
+ if (!plchan->at || !plchan->phychan) {
+ spin_unlock_irqrestore(&plchan->lock, flags);
+ return 0;
+ }
+
+ switch (cmd) {
+ case DMA_TERMINATE_ALL:
+ pl08x_stop_phy_chan(plchan->phychan);
+
+ /* Mark physical channel as free and free any slave signal */
+ if ((plchan->phychan->signal >= 0) && pl08x->pd->put_signal)
+ pl08x->pd->put_signal(plchan);
+ pl08x_put_phy_channel(pl08x, plchan->phychan);
+ plchan->phychan = NULL;
+
+ /* Dequeue jobs and free LLIs */
+ pl08x_free_txd(pl08x, plchan->at);
+ pl08x_free_txd_list(pl08x, &plchan->desc_list);
+
+ spin_unlock_irqrestore(&plchan->lock, flags);
+
+ return 0;
+ case DMA_PAUSE:
+ pl08x_pause_phy_chan(plchan->phychan);
+ spin_unlock_irqrestore(&plchan->lock, flags);
+ return 0;
+ case DMA_RESUME:
+ pl08x_resume_phy_chan(plchan->phychan);
+ spin_unlock_irqrestore(&plchan->lock, flags);
+ return 0;
+ default:
+ break;
+ }
+ spin_unlock_irqrestore(&plchan->lock, flags);
+
+ /* Unknown command */
+ return -ENXIO;
+}
+
+bool pl08x_filter_id(struct dma_chan *chan, void *chan_id)
+{
+ struct pl08x_dma_chan *plchan = to_pl08x_chan(chan);
+ char *name = chan_id;
+
+ /* Check that the channel is not taken! */
+ if (!strcmp(plchan->name, name))
+ return true;
+
+ return false;
+}
+
+struct dma_device dmac_memcpy = {
+ .device_alloc_chan_resources = pl08x_alloc_chan_resources,
+ .device_free_chan_resources = pl08x_free_chan_resources,
+ .device_prep_dma_memcpy = pl08x_prep_dma_memcpy,
+ .device_prep_dma_xor = NULL,
+ .device_prep_dma_memset = NULL,
+ .device_prep_dma_interrupt = pl08x_prep_dma_interrupt,
+ .device_tx_status = pl08x_dma_tx_status,
+ .device_issue_pending = pl08x_issue_pending,
+ .device_control = pl08x_control,
+ /*
+ * Align to 4-byte boundary
+ * This makes the DMAtests fail with grace on PB1176
+ * broken DMA hardware instead of locking everything
+ * up.
+ */
+ /* .copy_align = 2, */
+};
+
+struct dma_device dmac_slave = {
+ .device_alloc_chan_resources = pl08x_alloc_chan_resources,
+ .device_free_chan_resources = pl08x_free_chan_resources,
+ .device_prep_dma_xor = NULL,
+ .device_prep_dma_memset = NULL,
+ .device_prep_dma_interrupt = pl08x_prep_dma_interrupt,
+ .device_tx_status = pl08x_dma_tx_status,
+ .device_issue_pending = pl08x_issue_pending,
+ .device_prep_slave_sg = pl08x_prep_slave_sg,
+ .device_control = pl08x_control,
+};
+
+
+/*
+ * Just check that the device is there and active
+ * TODO: turn this bit on/off depending on the number of
+ * physical channels actually used, if it is zero... well
+ * shut it off.
+ */
+static void pl08x_ensure_on(struct pl08x_driver_data *pl08x)
+{
+ u32 val;
+
+ val = readl(pl08x->base + PL080_CONFIG);
+ val &= ~(PL080_CONFIG_M2_BE | PL080_CONFIG_M1_BE | PL080_CONFIG_ENABLE);
+ /* We implictly clear bit 1 and that means little-endian mode */
+ val |= PL080_CONFIG_ENABLE;
+ mb();
+ writel(val, pl08x->base + PL080_CONFIG);
+ mb();
+}
+
+/*
+ * Initialise the DMAC memcpy channels.
+ * Make a local wrapper to hold required data
+ */
+static int pl08x_dma_init_memcpy_channels(struct pl08x_driver_data *pl08x,
+ struct dma_device *memdev)
+{
+ struct pl08x_dma_chan *chan;
+ int i;
+
+ INIT_LIST_HEAD(&memdev->channels);
+ /*
+ * Register as many many memcpy as we have physical channels,
+ * we won't always be able to use all but the code will have
+ * to cope with that situation.
+ */
+ for (i = 0; i < pl08x->vd->channels; i++) {
+ chan = kzalloc(sizeof(struct pl08x_dma_chan), GFP_KERNEL);
+ if (!chan) {
+ dev_err(&pl08x->adev->dev,
+ "%s no memory for channel\n", __func__);
+ return -ENOMEM;
+ }
+
+ chan->host = pl08x;
+ chan->name = kasprintf(GFP_KERNEL, "memcpy%d", i);
+ if (!chan->name) {
+ kfree(chan);
+ return -ENOMEM;
+ }
+ chan->cd = &pl08x->pd->memcpy_channel;
+ dev_info(&pl08x->adev->dev,
+ "initialize virtual memcpy channel \"%s\"\n",
+ chan->name);
+
+ chan->chan.device = memdev;
+ atomic_set(&chan->last_issued, 0);
+ chan->lc = atomic_read(&chan->last_issued);
+
+ spin_lock_init(&chan->lock);
+ INIT_LIST_HEAD(&chan->desc_list);
+ tasklet_init(&chan->tasklet, pl08x_tasklet,
+ (unsigned long) chan);
+
+ list_add_tail(&chan->chan.device_node, &memdev->channels);
+ }
+ dev_info(&pl08x->adev->dev, "initialized %d virtual memcpy channels\n", i);
+ return i;
+}
+
+/*
+ * Initialise the DMAC slave channels.
+ * Make a local wrapper to hold required data
+ */
+static int pl08x_dma_init_slave_channels(struct pl08x_driver_data *pl08x,
+ struct dma_device *slave)
+{
+ struct pl08x_dma_chan *chan;
+ int i;
+
+ INIT_LIST_HEAD(&slave->channels);
+ for (i = 0; i < pl08x->pd->num_slave_channels; i++) {
+ chan = kzalloc(sizeof(struct pl08x_dma_chan), GFP_KERNEL);
+ if (!chan) {
+ dev_err(&pl08x->adev->dev,
+ "%s no memory for channel\n", __func__);
+ return -ENOMEM;
+ }
+
+ chan->host = pl08x;
+ chan->name = pl08x->pd->slave_channels[i].bus_id;
+ chan->cd = &pl08x->pd->slave_channels[i];
+ dev_info(&pl08x->adev->dev,
+ "initialize virtual channel \"%s\"\n",
+ chan->name);
+
+ chan->chan.device = slave;
+ atomic_set(&chan->last_issued, 0);
+ chan->lc = atomic_read(&chan->last_issued);
+
+ spin_lock_init(&chan->lock);
+ INIT_LIST_HEAD(&chan->desc_list);
+ tasklet_init(&chan->tasklet, pl08x_tasklet,
+ (unsigned long) chan);
+
+ list_add_tail(&chan->chan.device_node, &slave->channels);
+ }
+ dev_info(&pl08x->adev->dev, "initialized %d virtual slave channels\n", i);
+ return i;
+}
+
+static int pl08x_probe(struct amba_device *adev, struct amba_id *id)
+{
+ struct pl08x_driver_data *pl08x;
+ struct vendor_data *vd = id->data;
+ int ret = 0;
+ int i;
+
+ ret = amba_request_regions(adev, NULL);
+ if (ret)
+ return ret;
+
+ /* Create the driver state holder */
+ pl08x = kzalloc(sizeof(struct pl08x_driver_data), GFP_KERNEL);
+ if (!pl08x) {
+ ret = -ENOMEM;
+ goto out_no_pl08x;
+ }
+
+ /* Assign useful pointers to the driver state */
+ pl08x->adev = adev;
+ pl08x->vd = vd;
+
+ /* A DMA memory pool for LLIs, align on 1-byte boundary */
+ pl08x->pool = dma_pool_create(DRIVER_NAME, &pl08x->adev->dev,
+ PL08X_LLI_TSFR_SIZE, PL08X_ALIGN, 0);
+ if (!pl08x->pool) {
+ ret = -ENOMEM;
+ goto out_no_lli_pool;
+ }
+ pl08x->pool_ctr = 0;
+ pl08x->max_num_llis = 0;
+
+ spin_lock_init(&pl08x->lock);
+
+ pl08x->base = ioremap(adev->res.start, resource_size(&adev->res));
+ if (!pl08x->base) {
+ ret = -ENOMEM;
+ goto out_no_ioremap;
+ }
+
+ /* Turn on the PL08x */
+ pl08x_ensure_on(pl08x);
+
+ /*
+ * Attach the interrupt handler
+ */
+ writel(0x000000FF, pl08x->base + PL080_ERR_CLEAR);
+ writel(0x000000FF, pl08x->base + PL080_TC_CLEAR);
+ mb();
+
+ ret = request_irq(adev->irq[0], pl08x_irq, IRQF_DISABLED,
+ vd->name, pl08x);
+ if (ret) {
+ dev_err(&adev->dev, "%s failed to request "
+ "interrupt %d\n",
+ __func__, adev->irq[0]);
+ goto out_no_irq;
+ }
+
+ /* Initialize physical channels */
+ pl08x->phy_chans = kmalloc((vd->channels * sizeof(struct pl08x_phy_chan)),
+ GFP_KERNEL);
+ if (!pl08x->phy_chans) {
+ dev_err(&adev->dev, "%s failed to allocate "
+ "physical channel holders\n",
+ __func__);
+ goto out_no_phychans;
+ }
+
+ for (i = 0; i < vd->channels; i++) {
+ struct pl08x_phy_chan *ch = &pl08x->phy_chans[i];
+
+ ch->id = i;
+ ch->base = pl08x->base + PL080_Cx_BASE(i);
+ spin_lock_init(&ch->lock);
+ ch->serving = NULL;
+ ch->signal = -1;
+ dev_info(&adev->dev,
+ "physical channel %d is %s\n", i,
+ pl08x_phy_channel_busy(ch) ? "BUSY" : "FREE");
+ }
+
+ /* Get the platform data */
+ pl08x->pd = (struct pl08x_platform_data *)(adev->dev.platform_data);
+
+ /* Set caps */
+ dma_cap_set(DMA_MEMCPY, dmac_memcpy.cap_mask);
+ dma_cap_set(DMA_SLAVE, dmac_slave.cap_mask);
+ dmac_memcpy.dev = &adev->dev;
+ dmac_slave.dev = &adev->dev;
+
+ /* Register memcpy channels */
+ ret = pl08x_dma_init_memcpy_channels(pl08x, &dmac_memcpy);
+ if (ret <= 0) {
+ dev_warn(&pl08x->adev->dev,
+ "%s failed to enumerate memcpy channels - %d\n",
+ __func__, ret);
+ goto out_no_memcpy;
+ }
+ dmac_memcpy.chancnt = ret;
+
+ /* Register slave channels */
+ ret = pl08x_dma_init_slave_channels(pl08x, &dmac_slave);
+ if (ret <= 0) {
+ dev_warn(&pl08x->adev->dev,
+ "%s failed to enumerate slave channels - %d\n",
+ __func__, ret);
+ goto out_no_slave;
+ }
+ dmac_slave.chancnt = ret;
+
+ ret = dma_async_device_register(&dmac_memcpy);
+ if (ret) {
+ dev_warn(&pl08x->adev->dev,
+ "%s failed to register memcpy as an async device - %d\n",
+ __func__, ret);
+ goto out_no_memcpy_reg;
+ }
+
+ ret = dma_async_device_register(&dmac_slave);
+ if (ret) {
+ dev_warn(&pl08x->adev->dev,
+ "%s failed to register slave as an async device - %d\n",
+ __func__, ret);
+ goto out_no_slave_reg;
+ }
+
+ amba_set_drvdata(adev, pl08x);
+ dev_info(&pl08x->adev->dev, "ARM(R) %s DMA block initialized @%08x\n",
+ vd->name, adev->res.start);
+ return 0;
+
+out_no_slave_reg:
+ dma_async_device_unregister(&dmac_memcpy);
+out_no_memcpy_reg:
+ /* FIXME: free slave channels */
+out_no_slave:
+ /* FIXME: free memcpy channels */
+out_no_memcpy:
+ kfree(pl08x->phy_chans);
+out_no_phychans:
+ free_irq(adev->irq[0], pl08x);
+out_no_irq:
+ iounmap(pl08x->base);
+out_no_ioremap:
+ dma_pool_destroy(pl08x->pool);
+out_no_lli_pool:
+ kfree(pl08x);
+out_no_pl08x:
+ amba_release_regions(adev);
+ return ret;
+}
+
+/* PL080 has 8 channels and the PL080 have just 2 */
+static struct vendor_data vendor_pl080 = {
+ .name = "PL080",
+ .channels = 8,
+};
+
+static struct vendor_data vendor_pl081 = {
+ .name = "PL081",
+ .channels = 2,
+};
+
+static struct amba_id pl08x_ids[] = {
+ /* PL080 */
+ {
+ .id = 0x00041080,
+ .mask = 0x000fffff,
+ .data = &vendor_pl080,
+ },
+ /* PL081 */
+ {
+ .id = 0x00041081,
+ .mask = 0x000fffff,
+ .data = &vendor_pl081,
+ },
+ /* Nomadik 8815 PL080 variant */
+ {
+ .id = 0x00280880,
+ .mask = 0x00ffffff,
+ .data = &vendor_pl080,
+ },
+ { 0, 0 },
+};
+
+static struct amba_driver pl08x_amba_driver = {
+ .drv.name = DRIVER_NAME,
+ .id_table = pl08x_ids,
+ .probe = pl08x_probe,
+};
+
+static int __init pl08x_init(void)
+{
+ int retval;
+ retval = amba_driver_register(&pl08x_amba_driver);
+ if (retval)
+ printk(KERN_WARNING
+ "PL08X::pl08x_init() - failed to register as an amba device - %d\n",
+ retval);
+ return retval;
+}
+subsys_initcall(pl08x_init);
diff --git a/include/linux/amba/pl08x.h b/include/linux/amba/pl08x.h
new file mode 100644
index 0000000..5f9b16f
--- /dev/null
+++ b/include/linux/amba/pl08x.h
@@ -0,0 +1,173 @@
+/*
+ * linux/amba/pl08x.h - ARM PrimeCell DMA Controller driver
+ *
+ * Copyright (C) 2005 ARM Ltd
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * pl08x information required by platform code
+ *
+ * Please credit ARM.com
+ * Documentation: ARM DDI 0196D
+ *
+ */
+
+#ifndef AMBA_PL08X_H
+#define AMBA_PL08X_H
+
+/* We need sizes of structs from this header */
+#include <linux/dmaengine.h>
+
+/**
+ * struct pl08x_channel_data - data structure to pass info between
+ * platform and PL08x driver regarding channel configuration
+ * @bus_id: name of this device channel, not just a device name since
+ * devices may have more than one channel e.g. "foo_tx"
+ * @min_signal: the minimum DMA signal number to be muxed in for this
+ * channel (for platforms supporting muxed signals). If you have
+ * static assignments, make sure this is set to the assigned signal
+ * number, PL08x have 16 possible signals in number 0 thru 15 so
+ * when these are not enough they often get muxed (in hardware)
+ * disabling simultaneous use of the same channel for two devices.
+ * @max_signal: the maximum DMA signal number to be muxed in for
+ * the channel. Set to the same as min_signal for
+ * devices with static assignments
+ * @muxval: a number usually used to poke into some mux regiser to
+ * mux in the signal to this channel
+ * @cctl_opt: default options for the channel control register
+ * @circular_buffer: whether the buffer passed in is circular and
+ * shall simply be looped round round (like a record baby round
+ * round round round)
+ */
+struct pl08x_channel_data {
+ char *bus_id;
+ int min_signal;
+ int max_signal;
+ u32 muxval;
+ unsigned int cctl; /* Turn me into u32? */
+ u32 ccfg;
+ bool circular_buffer;
+};
+
+/**
+ * struct pl08x_bus_data - information of source or destination
+ * busses for a transfer
+ * @addr: current address
+ * @maxwidth: the maximum width of a transfer on this bus
+ * @buswidth: the width of this bus in bytes: 1, 2 or 4
+ * @fill_bytes: bytes required to fill to the next bus memory
+ * boundary
+ */
+struct pl08x_bus_data {
+ dma_addr_t addr;
+ u8 maxwidth;
+ u8 buswidth;
+ u32 fill_bytes;
+};
+
+/**
+ * struct pl08x_phy_chan - holder for the physical channels
+ * @id: physical index to this channel
+ * @lock: a lock to use when altering an instance of this struct
+ * @signal: the physical signal (aka channel) serving this
+ * physical channel right now
+ * @serving: the virtual channel currently being served by this
+ * physical channel
+ */
+struct pl08x_phy_chan {
+ unsigned int id;
+ void __iomem *base;
+ spinlock_t lock;
+ int signal;
+ struct pl08x_dma_chan *serving;
+ u32 csrc;
+ u32 cdst;
+ u32 clli;
+ u32 cctl;
+ u32 ccfg;
+};
+
+/**
+ * struct pl08x_txd - wrapper for struct dma_async_tx_descriptor
+ * @llis_bus: DMA memory address (physical) start for the LLIs
+ * @llis_va: virtual memory address start for the LLIs
+ */
+struct pl08x_txd {
+ struct dma_async_tx_descriptor tx;
+ enum dma_data_direction direction;
+ struct pl08x_bus_data srcbus;
+ struct pl08x_bus_data dstbus;
+ int len;
+ dma_addr_t llis_bus;
+ void *llis_va;
+ struct list_head node;
+ struct pl08x_channel_data *cd;
+ bool active;
+ /* Settings to be put into the physical channel when we submit this txd */
+ u32 csrc;
+ u32 cdst;
+ u32 clli;
+ u32 cctl;
+};
+
+/**
+ * struct pl08x_dma_chan - this structure wraps a DMA ENGINE channel
+ * @chan: wrappped abstract channel
+ * @phychan: the physical channel utilized by this channel, if there is one
+ * @tasklet: tasklet scheduled by the IRQ to handle actual work etc
+ * @name: name of channel
+ * @cd: channel platform data
+ * @amba_addr: address for RX/TX according to the PrimeCell config
+ * @amba_direction: current direction of this channel according to
+ * @lc: last completed transaction on this channel
+ * @desc_list: queued transactions pending on this channel
+ * @at: active transaction on this channel
+ * @lock: a lock for this channel data
+ * @host: a pointer to the host (internal use)
+ */
+struct pl08x_dma_chan {
+ struct dma_chan chan;
+ struct pl08x_phy_chan *phychan;
+ struct tasklet_struct tasklet;
+ char *name;
+ struct pl08x_channel_data *cd;
+ dma_addr_t amba_addr;
+ enum dma_data_direction amba_direction;
+ atomic_t last_issued;
+ dma_cookie_t lc;
+ struct list_head desc_list;
+ struct pl08x_txd *at;
+ spinlock_t lock;
+ void *host;
+};
+
+/**
+ * struct pl08x_platform_data - the platform configuration for the
+ * PL08x PrimeCells.
+ * @slave_channels: the channels defined for the different devices on the
+ * platform, all inclusive, including multiplexed channels. The available
+ * physical channels will be multiplexed around these signals as they
+ * are requested, just enumerate all possible channels.
+ * @get_signal: request a physical signal to be used for a DMA
+ * transfer immediately: if there is some multiplexing or similar blocking
+ * the use of the channel the transfer can be denied by returning
+ * less than zero, else it returns the allocated signal number
+ * @put_signal: indicate to the platform that this physical signal is not
+ * running any DMA transfer and multiplexing can be recycled
+ * @bus_bit_lli: Bit[0] of the address indicated which AHB bus master the
+ * LLI addresses are on 0/1 Master 1/2.
+ */
+struct pl08x_platform_data {
+ struct pl08x_channel_data *slave_channels;
+ unsigned int num_slave_channels;
+ struct pl08x_channel_data memcpy_channel;
+ int (*get_signal)(struct pl08x_dma_chan *);
+ void (*put_signal)(struct pl08x_dma_chan *);
+ unsigned int bus_bit_lli:1;
+};
+
+bool pl08x_filter_id(struct dma_chan *chan, void *chan_id);
+
+#endif /* AMBA_PL08X_H */
--
1.6.3.3
Linus,
I haven't reviewed it completely. Will do it in some time.
On 6/11/2010 8:57 PM, Linus WALLEIJ wrote:
[snip...]
> diff --git a/drivers/dma/amba-pl08x.c b/drivers/dma/amba-pl08x.c
> new file mode 100644
> index 0000000..50531fa
> --- /dev/null
> +++ b/drivers/dma/amba-pl08x.c
> @@ -0,0 +1,1925 @@
> +/*
> + * Copyright (c) 2006 ARM Ltd.
> + * Copyright (c) 2010 ST-Ericsson SA
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License as published by the Free
> + * Software Foundation; either version 2 of the License, or (at your option)
> + * any later version.
> + *
> + * This program is distributed in the hope that it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program; if not, write to the Free Software Foundation, Inc., 59
> + * Temple Place - Suite 330, Boston, MA 02111-1307, USA.
> + *
> + * The full GNU General Public License is iin this distribution in the
> + * file called COPYING.
> + *
> + * Documentation: ARM DDI 0196G == PL080
> + * Documentation: ARM DDI 0218E == PL081
> + *
> + * PL080 & PL081 both have 16 sets of DMA signals. They differ in the number
> + * of channels which may be in use at once. Also PL080 has a dual bus master,
> + * PL081 has a single master.
> + *
> + * Memory to peripheral transfer may be visualized as
> + * Get data from memory to DMAC
> + * Until no data left
> + * On burst request from peripheral
> + * Destination burst from DMAC to peripheral
> + * Clear burst request
> + * Raise terminal count interrupt
> + *
> + * For peripherals with a FIFO:
> + * Source burst size == half the depth of the peripheral FIFO
> + * Destination burst size == width of the peripheral FIFO
> + *
I didn't get it completely, why burst depends upon width of peripheral FIFO.
> + * (Bursts are irrelevant for mem to mem transfers - there are no burst
> + * signals)
I agree that there are no request lines from memories but still we can program
them with burst in order to faster the transfer. This burst feature is
automatically handled by DMA.
> + *
> + * ASSUMES default (little) endianness for DMA transfers
> + *
> + * Only DMAC flow control is implemented
> + *
> + * Global TODO:
> + * - Break out common code from arch/arm/mach-s3c64xx and share
> + */
> +#include <linux/device.h>
> +#include <linux/init.h>
> +#include <linux/module.h>
> +#include <linux/pci.h>
> +#include <linux/interrupt.h>
> +#include <linux/slab.h>
> +#include <linux/dmapool.h>
> +#include <linux/amba/bus.h>
> +#include <linux/dmaengine.h>
> +#include <linux/amba/pl08x.h>
> +#include <linux/amba/dma.h>
> +
> +#include <asm/hardware/pl080.h>
> +#include <asm/dma.h>
> +#include <asm/mach/dma.h>
> +#include <asm/atomic.h>
> +#include <asm/processor.h>
> +#include <asm/cacheflush.h>
> +
> +#define DRIVER_NAME "pl08xdmac"
> +
> +/**
> + * struct vendor_data - vendor-specific config parameters
> + * for PL08x derivates
> + * @name: the name of this specific variant
> + * @channels: the number of channels available in this variant
> + */
> +struct vendor_data {
> + char *name;
> + u8 channels;
> +};
> +
> +/*
> + * PL08X private data structures
> + * An LLI struct - see pl08x TRM
> + * Note that next uses bit[0] as a bus bit,
> + * start & end do not - their bus bit info
> + * is in cctl
> + */
> +struct lli {
> + dma_addr_t src;
> + dma_addr_t dst;
> + dma_addr_t next;
> + u32 cctl;
> +};
> +
> +/**
> + * struct pl08x_driver_data - the local state holder for the PL08x
> + * @base: virtual memory base (remapped) for the PL08x
> + * @adev: the corresponding AMBA (PrimeCell) bus entry
> + * @vd: vendor data for this PL08x variant
> + * @pd: platform data passed in from the platform/machine
> + * @phy_chans: array of data for the physical channels
> + * @pool: a pool for the LLI descriptors
> + * @pool_ctr: counter of LLIs in the pool
> + * @max_num_llis: maximum number of LLIs, i.e. longest linked transfer
> + * length, submitted so far
What is the significance of this field? What it is used for?
> + * @lock: a spinlock for this struct
> + */
> +struct pl08x_driver_data {
> + void __iomem *base;
> + struct amba_device *adev;
> + struct vendor_data *vd;
> + struct pl08x_platform_data *pd;
> + struct pl08x_phy_chan *phy_chans;
> + struct dma_pool *pool;
> + int pool_ctr;
> + int max_num_llis;
> + spinlock_t lock;
> +};
> +
> +#ifdef MODULE
> +
> +# error "AMBA PL08X DMA CANNOT BE COMPILED AS A LOADABLE MODULE AT PRESENT"
> +
> +/*
> + a) Some devices might make use of DMA during boot
> + (esp true for DMAENGINE implementation)
> + b) Memory allocation will need much more attention
> + before load/unload can be supported
> + */
> +#endif
> +
> +/*
> + * PL08X specific defines
> + */
> +
> +/*
> + * Memory boundaries: the manual for PL08x says that the controller
> + * cannot read past a 1KiB boundary, so these defines are used to
> + * create transfer LLIs that do not cross such boundaries.
> + */
> +#define PL08X_BOUNDARY_SHIFT (10) /* 1KB 0x400 */
> +#define PL08X_BOUNDARY_SIZE (1 << PL08X_BOUNDARY_SHIFT)
> +
> +/* Minimum period between work queue runs */
> +#define PL08X_WQ_PERIODMIN 20
> +
> +/* Size (bytes) of each LLI buffer allocated for one transfer */
> +# define PL08X_LLI_TSFR_SIZE 0x2000
> +
> +/* Maximimum times we call dma_pool_alloc on this pool without freeing */
> +#define PL08X_MAX_ALLOCS 0x40
> +#define MAX_NUM_TSFR_LLIS (PL08X_LLI_TSFR_SIZE/sizeof(struct lli))
> +#define PL08X_ALIGN 8
> +
> +static inline struct pl08x_dma_chan *to_pl08x_chan(struct dma_chan *chan)
> +{
> + return container_of(chan, struct pl08x_dma_chan, chan);
> +}
> +
> +/*
> + * Physical channel handling
> + */
> +
> +/* Whether a certain channel is busy or not */
> +static int pl08x_phy_channel_busy(struct pl08x_phy_chan *ch)
> +{
> + unsigned int val;
> +
> + val = readl(ch->base + PL080_CH_CONFIG);
> + return val & PL080_CONFIG_ACTIVE;
> +}
> +
> +/*
> + * Set the initial DMA register values i.e. those for the first LLI
> + * The next lli pointer and the configuration interrupt bit have
> + * been set when the LLIs were constructed
> + */
> +static void pl08x_set_cregs(struct pl08x_driver_data *pl08x,
> + struct pl08x_phy_chan *ch)
> +{
> + u32 val;
> +
> + /* Wait for channel inactive */
> + val = readl(ch->base + PL080_CH_CONFIG);
> + while (val & PL080_CONFIG_ACTIVE)
> + val = readl(ch->base + PL080_CH_CONFIG);
can we use pl08x_phy_channel_busy() instead of above code?
> +
> + dev_vdbg(&pl08x->adev->dev,
> + "WRITE channel %d: csrc=%08x, cdst=%08x, "
> + "cctl=%08x, clli=%08x, ccfg=%08x\n",
> + ch->id,
> + ch->csrc,
> + ch->cdst,
> + ch->cctl,
> + ch->clli,
> + ch->ccfg);
> +
> + writel(ch->csrc, ch->base + PL080_CH_SRC_ADDR);
> + writel(ch->cdst, ch->base + PL080_CH_DST_ADDR);
> + writel(ch->clli, ch->base + PL080_CH_LLI);
> + writel(ch->cctl, ch->base + PL080_CH_CONTROL);
> + writel(ch->ccfg, ch->base + PL080_CH_CONFIG);
> + mb();
> +}
> +
> +static inline void pl08x_config_phychan_for_txd(struct pl08x_dma_chan *plchan)
> +{
> + struct pl08x_channel_data *cd = plchan->cd;
> + struct pl08x_phy_chan *phychan = plchan->phychan;
> + struct pl08x_txd *txd = plchan->at;
> +
> + /* Copy the basic control register calculated at transfer config */
> + phychan->csrc = txd->csrc;
> + phychan->cdst = txd->cdst;
> + phychan->clli = txd->clli;
> + phychan->cctl = txd->cctl;
> +
> + /* Assign the signal to the proper control registers */
> + phychan->ccfg = cd->ccfg;
> + phychan->ccfg &= ~PL080_CONFIG_SRC_SEL_MASK;
> + phychan->ccfg &= ~PL080_CONFIG_DST_SEL_MASK;
> + /* If it wasn't set from AMBA, ignore it */
> + if (txd->direction == DMA_TO_DEVICE)
> + /* Select signal as destination */
> + phychan->ccfg |=
> + (phychan->signal << PL080_CONFIG_DST_SEL_SHIFT);
> + else if (txd->direction == DMA_FROM_DEVICE)
> + /* Select signal as source */
> + phychan->ccfg |=
> + (phychan->signal << PL080_CONFIG_SRC_SEL_SHIFT);
> + /* Always enable error interrupts */
> + phychan->ccfg |= PL080_CONFIG_ERR_IRQ_MASK;
> + /* Always enable terminal interrupts */
> + phychan->ccfg |= PL080_CONFIG_TC_IRQ_MASK;
> +}
> +
> +/*
> + * Enable the DMA channel
> + * Assumes all other configuration bits have been set
> + * as desired before this code is called
> + */
> +static void pl08x_enable_phy_chan(struct pl08x_driver_data *pl08x,
> + struct pl08x_phy_chan *ch)
> +{
> + u32 val;
> +
> + /*
> + * Do not access config register until channel shows as disabled
> + */
> + while (readl(pl08x->base + PL080_EN_CHAN) & (1 << ch->id))
> + ;
> +
> + /*
> + * Do not access config register until channel shows as inactive
> + */
> + val = readl(ch->base + PL080_CH_CONFIG);
> + while ((val & PL080_CONFIG_ACTIVE) || (val & PL080_CONFIG_ENABLE))
> + val = readl(ch->base + PL080_CH_CONFIG);
above 3 fns are always called in order, i.e. pl08x_enable_phy_chan will
be called after pl08x_set_cregs, so we may not require these checks
here. Is my understanding correct??
> +
> + writel(val | PL080_CONFIG_ENABLE, ch->base + PL080_CH_CONFIG);
> + mb();
> +}
> +
> +/*
> + * Overall DMAC remains enabled always.
> + *
> + * Disabling individual channels could lose data.
> + *
> + * Disable the peripheral DMA after disabling the DMAC
> + * in order to allow the DMAC FIFO to drain, and
> + * hence allow the channel to show inactive
> + *
> + */
> +static void pl08x_pause_phy_chan(struct pl08x_phy_chan *ch)
> +{
> + u32 val;
> +
> + /* Set the HALT bit and wait for the FIFO to drain */
> + val = readl(ch->base + PL080_CH_CONFIG);
> + val |= PL080_CONFIG_HALT;
> + writel(val, ch->base + PL080_CH_CONFIG);
> +
> + /* Wait for channel inactive */
> + val = readl(ch->base + PL080_CH_CONFIG);
> + while (val & PL080_CONFIG_ACTIVE)
> + val = readl(ch->base + PL080_CH_CONFIG);
can we use pl08x_phy_channel_busy() instead of above code?
Please check everywhere.
> +
> + mb();
> +
> + return;
return not required!!
> +}
> +
> +static void pl08x_resume_phy_chan(struct pl08x_phy_chan *ch)
can these small fns be made inline???
> +{
> + u32 val;
> +
> + /* Clear the HALT bit */
> + val = readl(ch->base + PL080_CH_CONFIG);
> + val &= ~PL080_CONFIG_HALT;
> + writel(val, ch->base + PL080_CH_CONFIG);
> + mb();
> +
> + return;
> +}
> +
> +
> +/* Stops the channel */
> +static void pl08x_stop_phy_chan(struct pl08x_phy_chan *ch)
> +{
> + u32 val;
> +
> + pl08x_pause_phy_chan(ch);
> +
> + /* Disable channel */
> + val = readl(ch->base + PL080_CH_CONFIG);
> + val &= ~PL080_CONFIG_ENABLE;
> + writel(val, ch->base + PL080_CH_CONFIG);
> + mb();
> +
> + return;
same here. return not required.
> +}
> +
> +static inline u32 get_bytes_in_cctl(u32 cctl)
> +{
> + /* The source width defines the number of bytes */
> + u32 bytes = cctl & PL080_CONTROL_TRANSFER_SIZE_MASK;
> +
> + switch ((cctl >> 18) & 3) {
better to use Macros instead of magic numbers here!!!
> + case PL080_WIDTH_8BIT:
> + break;
> + case PL080_WIDTH_16BIT:
> + bytes *= 2;
> + break;
> + case PL080_WIDTH_32BIT:
> + bytes *= 4;
> + break;
> + }
> + return bytes;
> +}
> +
> +static u32 pl08x_getbytes_phy_chan(struct pl08x_phy_chan *ch)
> +{
> + u32 bytes;
> +
> + /* FIXME: follow all queued transactions */
> + bytes = get_bytes_in_cctl(readl(ch->base + PL080_CH_CONTROL));
> + /* TODO: follow the LLI to see the sum summarum */
> + return bytes;
> +}
> +
> +/*
> + * Allocate a physical channel for a virtual channel
> + */
> +static struct pl08x_phy_chan *
> +pl08x_get_phy_channel(struct pl08x_driver_data *pl08x,
> + struct pl08x_dma_chan *virt_chan)
> +{
> + struct pl08x_phy_chan *ch = NULL;
> + unsigned long flags;
> + int i;
> +
> + /*
> + * Try to locate a physical channel to be used for
> + * this transfer. If all are taken return NULL and
> + * the requester will have to cope by using some fallback
> + * PIO mode or retrying later.
> + */
> + for (i = 0; i < pl08x->vd->channels; i++) {
> + ch = &pl08x->phy_chans[i];
> +
> + spin_lock_irqsave(&ch->lock, flags);
> +
> + if (!ch->serving) {
> + ch->serving = virt_chan;
> + ch->signal = -1;
> + spin_unlock_irqrestore(&ch->lock, flags);
> + break;
> + }
> +
> + spin_unlock_irqrestore(&ch->lock, flags);
> + }
> +
> + if (i == pl08x->vd->channels) {
> + /* No physical channel available, cope with it */
> + return NULL;
> + }
> +
> + return ch;
> +}
> +
> +static inline void pl08x_put_phy_channel(struct pl08x_driver_data *pl08x,
> + struct pl08x_phy_chan *ch)
> +{
> + unsigned long flags;
> +
> + /* Stop the channel and clear its interrupts */
> + pl08x_stop_phy_chan(ch);
> + writel((1 << ch->id), pl08x->base + PL080_ERR_CLEAR);
> + writel((1 << ch->id), pl08x->base + PL080_TC_CLEAR);
> +
> + /* Mark it as free */
> + spin_lock_irqsave(&ch->lock, flags);
> + ch->serving = NULL;
> + ch->signal = -1;
> + spin_unlock_irqrestore(&ch->lock, flags);
> +}
> +
> +/*
> + * LLI handling
> + */
> +
> +static inline unsigned int pl08x_get_bytes_for_cctl(unsigned int coded)
> +{
> + switch (coded) {
> + case PL080_WIDTH_8BIT:
> + return 1;
> + case PL080_WIDTH_16BIT:
> + return 2;
> + case PL080_WIDTH_32BIT:
> + return 4;
> + default:
> + break;
> + }
> + BUG();
> + return 0;
> +}
> +
> +static inline u32 pl08x_cctl_bits(u32 cctl,
> + u8 srcwidth,
> + u8 dstwidth,
> + u32 tsize)
Not sure, if we should write above function prototype in just 2 lines,
or is it okay to write it the way it is written.
> +{
> + u32 retbits = cctl;
> +
> + /* Remove all src, dst and transfersize bits */
> + retbits &= ~PL080_CONTROL_DWIDTH_MASK;
> + retbits &= ~PL080_CONTROL_SWIDTH_MASK;
> + retbits &= ~PL080_CONTROL_TRANSFER_SIZE_MASK;
> +
> + /* Then set the bits according to the parameters */
> + switch(srcwidth) {
> + case 1:
> + retbits |= PL080_WIDTH_8BIT << PL080_CONTROL_SWIDTH_SHIFT;
> + break;
> + case 2:
> + retbits |= PL080_WIDTH_16BIT << PL080_CONTROL_SWIDTH_SHIFT;
> + break;
> + case 4:
> + retbits |= PL080_WIDTH_32BIT << PL080_CONTROL_SWIDTH_SHIFT;
> + break;
> + default:
> + BUG();
> + break;
> + }
> +
> + switch(dstwidth) {
> + case 1:
> + retbits |= PL080_WIDTH_8BIT << PL080_CONTROL_DWIDTH_SHIFT;
> + break;
> + case 2:
> + retbits |= PL080_WIDTH_16BIT << PL080_CONTROL_DWIDTH_SHIFT;
> + break;
> + case 4:
> + retbits |= PL080_WIDTH_32BIT << PL080_CONTROL_DWIDTH_SHIFT;
> + break;
> + default:
> + BUG();
> + break;
> + }
> +
> + retbits |= tsize << PL080_CONTROL_TRANSFER_SIZE_SHIFT;
> + return retbits;
> +}
> +
> +/*
> + * Autoselect a master bus to use for the transfer
> + * this prefers the destination bus if both available
> + * if fixed address on one bus the other will be chosen
> + */
> +void pl08x_choose_master_bus(struct pl08x_bus_data *src_bus,
> + struct pl08x_bus_data *dst_bus, struct pl08x_bus_data **mbus,
> + struct pl08x_bus_data **sbus, u32 cctl)
> +{
> + if (!cctl & PL080_CONTROL_DST_INCR) {
> + *mbus = src_bus;
> + *sbus = dst_bus;
> + } else if (!cctl & PL080_CONTROL_SRC_INCR) {
> + *mbus = dst_bus;
> + *sbus = src_bus;
> + } else {
> + if (dst_bus->buswidth == 4) {
> + *mbus = dst_bus;
> + *sbus = src_bus;
> + } else if (src_bus->buswidth == 4) {
> + *mbus = src_bus;
> + *sbus = dst_bus;
> + } else if (dst_bus->buswidth == 2) {
> + *mbus = dst_bus;
> + *sbus = src_bus;
> + } else if (src_bus->buswidth == 2) {
> + *mbus = src_bus;
> + *sbus = dst_bus;
> + } else {
> + /* src_bus->buswidth == 1 */
> + *mbus = dst_bus;
> + *sbus = src_bus;
> + }
> + }
> +}
> +
> +/*
> + * Fills in one LLI for a certain transfer descriptor
> + * and advance the counter
> + */
> +int pl08x_fill_lli_for_desc(struct pl08x_driver_data *pl08x,
> + struct pl08x_txd *txd, int num_llis, int len,
> + u32 cctl, u32 *remainder)
> +{
> + struct lli *llis_va = (struct lli *)(txd->llis_va);
> + struct lli *llis_bus = (struct lli *)(txd->llis_bus);
> +
> + BUG_ON(num_llis >= MAX_NUM_TSFR_LLIS);
> +
> + llis_va[num_llis].cctl = cctl;
> + llis_va[num_llis].src = txd->srcbus.addr;
> + llis_va[num_llis].dst = txd->dstbus.addr;
> + /*
> + * The bus bit is added to the next lli's address
> + */
> + llis_va[num_llis].next =
> + (dma_addr_t)((u32) &(llis_bus[num_llis + 1])
> + | pl08x->pd->bus_bit_lli);
> +
> + if (cctl & PL080_CONTROL_SRC_INCR)
> + txd->srcbus.addr += len;
> + if (cctl & PL080_CONTROL_DST_INCR)
> + txd->dstbus.addr += len;
> +
> + *remainder -= len;
> +
> + return num_llis + 1;
> +}
> +
> +/*
> + * Return number of bytes to fill to boundary, or len
> + */
> +static inline u32 pl08x_pre_boundary(u32 addr, u32 len)
> +{
> + u32 boundary;
> +
> + boundary = ((addr >> PL08X_BOUNDARY_SHIFT) + 1)
> + << PL08X_BOUNDARY_SHIFT;
> +
> + if (boundary < addr + len)
> + return boundary - addr;
> + else
> + return len;
> +}
> +
> +/*
> + * This fills in the table of LLIs for the transfer descriptor
> + * Note that we assume we never have to change the burst sizes
> + * Return 0 for error
> + */
> +static int pl08x_fill_llis_for_desc(struct pl08x_driver_data *pl08x,
> + struct pl08x_txd *txd)
> +{
> + struct pl08x_channel_data *cd = txd->cd;
> + struct pl08x_bus_data *mbus, *sbus;
> + u32 remainder;
> + int num_llis = 0;
> + u32 cctl;
> + int max_bytes_per_lli;
> + int total_bytes = 0;
> + struct lli *llis_va;
> + struct lli *llis_bus;
> +
> + if (!txd) {
> + dev_err(&pl08x->adev->dev, "%s no descriptor\n", __func__);
> + return 0;
> + }
> +
> + txd->llis_va = dma_pool_alloc(pl08x->pool, GFP_KERNEL,
> + &txd->llis_bus);
> + if (!txd->llis_va) {
> + dev_err(&pl08x->adev->dev, "%s no memory for llis\n", __func__);
> + return 0;
> + }
> +
> + pl08x->pool_ctr++;
> +
> + /*
> + * Initialize bus values for this transfer
> + * from the passed optimal values
> + */
> + if (!cd) {
> + dev_err(&pl08x->adev->dev, "%s no channel data\n", __func__);
> + return 0;
> + }
> +
> + /* Get the default CCTL from the platform data */
> + cctl = cd->cctl;
> +
> + /* Find maximum width of the source bus */
> + txd->srcbus.maxwidth =
> + pl08x_get_bytes_for_cctl((cctl & PL080_CONTROL_SWIDTH_MASK) >>
> + PL080_CONTROL_SWIDTH_SHIFT);
> +
> + /* Find maximum width of the destination bus */
> + txd->dstbus.maxwidth =
> + pl08x_get_bytes_for_cctl((cctl & PL080_CONTROL_DWIDTH_MASK) >>
> + PL080_CONTROL_DWIDTH_SHIFT);
> +
> + /* Set up the bus widths to the maximum */
> + txd->srcbus.buswidth = txd->srcbus.maxwidth;
> + txd->dstbus.buswidth = txd->dstbus.maxwidth;
> + dev_vdbg(&pl08x->adev->dev,
> + "%s source bus is %d bytes wide, dest bus is %d bytes wide\n",
> + __func__, txd->srcbus.buswidth, txd->dstbus.buswidth);
> +
> +
> + /*
> + * bytes transferred == tsize * MIN(buswidths), not max(buswidths)
> + */
> + max_bytes_per_lli = min(txd->srcbus.buswidth, txd->dstbus.buswidth) *
> + PL080_CONTROL_TRANSFER_SIZE_MASK;
> + dev_vdbg(&pl08x->adev->dev,
> + "%s max bytes per lli = %d\n",
> + __func__, max_bytes_per_lli);
> +
> + /* We need to count this down to zero */
> + remainder = txd->len;
> + dev_vdbg(&pl08x->adev->dev,
> + "%s remainder = %d\n",
> + __func__, remainder);
> +
> + /*
> + * Choose bus to align to
> + * - prefers destination bus if both available
> + * - if fixed address on one bus chooses other
> + */
> + pl08x_choose_master_bus(&txd->srcbus,
> + &txd->dstbus, &mbus, &sbus, cctl);
> +
> + if (txd->len < mbus->buswidth) {
> + /*
> + * Less than a bus width available
> + * - send as single bytes
> + */
> + while (remainder) {
> + dev_vdbg(&pl08x->adev->dev,
> + "%s single byte LLIs for a transfer of less than a bus width (remain %08x)\n",
> + __func__, remainder);
> + cctl = pl08x_cctl_bits(cctl, 1, 1, 1);
> + num_llis =
> + pl08x_fill_lli_for_desc(pl08x, txd, num_llis, 1,
> + cctl, &remainder);
> + total_bytes++;
> + }
> + } else {
> + /*
> + * Make one byte LLIs until master bus is aligned
> + * - slave will then be aligned also
> + */
> + while ((mbus->addr) % (mbus->buswidth)) {
> + dev_vdbg(&pl08x->adev->dev,
> + "%s adjustment lli for less than bus width (remain %08x)\n",
> + __func__, remainder);
> + cctl = pl08x_cctl_bits(cctl, 1, 1, 1);
> + num_llis = pl08x_fill_lli_for_desc
> + (pl08x, txd, num_llis, 1, cctl, &remainder);
> + total_bytes++;
> + }
> +
> + /*
> + * Master now aligned
> + * - if slave is not then we must set its width down
> + */
> + if (sbus->addr % sbus->buswidth) {
> + dev_dbg(&pl08x->adev->dev,
> + "%s set down bus width to one byte\n",
> + __func__);
> +
> + sbus->buswidth = 1;
> + }
> +
> + /*
> + * Make largest possible LLIs until less than one bus width left
> + */
> + while (remainder > (mbus->buswidth - 1)) {
> + int lli_len, target_len;
> + int tsize;
> + int odd_bytes;
> +
> + /*
> + * If enough left try to send max possible,
> + * otherwise try to send the remainder
> + */
> + target_len = remainder;
> + if (remainder > max_bytes_per_lli)
> + target_len = max_bytes_per_lli;
> +
> + /*
> + * Set bus lengths for incrementing busses
> + * to number of bytes which fill to next memory
> + * boundary
> + */
> + if (cctl & PL080_CONTROL_SRC_INCR)
> + txd->srcbus.fill_bytes =
> + pl08x_pre_boundary(
> + txd->srcbus.addr,
> + remainder);
> + else
> + txd->srcbus.fill_bytes =
> + max_bytes_per_lli;
> +
> + if (cctl & PL080_CONTROL_DST_INCR)
> + txd->dstbus.fill_bytes =
> + pl08x_pre_boundary(
> + txd->dstbus.addr,
> + remainder);
> + else
> + txd->dstbus.fill_bytes =
> + max_bytes_per_lli;
> +
> + /*
> + * Find the nearest
> + */
> + lli_len = min(txd->srcbus.fill_bytes,
> + txd->dstbus.fill_bytes);
> +
> + BUG_ON(lli_len > remainder);
> +
> + if (lli_len <= 0) {
> + dev_err(&pl08x->adev->dev,
> + "%s lli_len is %d, <= 0\n",
> + __func__, lli_len);
> + return 0;
> + }
> +
> + if (lli_len == target_len) {
> + /*
> + * Can send what we wanted
> + */
> + /*
> + * Maintain alignment
> + */
> + lli_len = (lli_len/mbus->buswidth) *
> + mbus->buswidth;
> + odd_bytes = 0;
> + } else {
> + /*
> + * So now we know how many bytes to transfer
> + * to get to the nearest boundary
> + * The next lli will past the boundary
> + * - however we may be working to a boundary
> + * on the slave bus
> + * We need to ensure the master stays aligned
> + */
> + odd_bytes = lli_len % mbus->buswidth;
> + /*
> + * - and that we are working in multiples
> + * of the bus widths
> + */
> + lli_len -= odd_bytes;
> +
> + }
> +
> + if (lli_len) {
> + /*
> + * Check against minimum bus alignment:
> + * Calculate actual transfer size in relation to bus
> + * width an get a maximum remainder of the smallest
> + * bus width - 1
> + */
> + /* FIXME: use round_down()? */
> + tsize = lli_len / min(mbus->buswidth, sbus->buswidth);
> + lli_len = tsize * min(mbus->buswidth, sbus->buswidth);
> +
> + if (target_len != lli_len) {
> + dev_vdbg(&pl08x->adev->dev,
> + "%s can't send what we want. Desired %08x, lli of %08x bytes in txd of %08x\n",
> + __func__, target_len, lli_len, txd->len);
> + }
> +
> + cctl = pl08x_cctl_bits(cctl,
> + txd->srcbus.buswidth,
> + txd->dstbus.buswidth,
> + tsize);
> +
> + dev_vdbg(&pl08x->adev->dev,
> + "%s fill lli with single lli chunk of size %08x (remainder %08x)\n",
> + __func__, lli_len, remainder);
> + num_llis = pl08x_fill_lli_for_desc(pl08x, txd,
> + num_llis, lli_len, cctl,
> + &remainder);
> + total_bytes += lli_len;
> + }
> +
> +
> + if (odd_bytes) {
> + /*
> + * Creep past the boundary,
> + * maintaining master alignment
> + */
> + int j;
> + for (j = 0; (j < mbus->buswidth)
> + && (remainder); j++) {
> + cctl = pl08x_cctl_bits(cctl, 1, 1, 1);
> + dev_vdbg(&pl08x->adev->dev,
> + "%s align with boundardy, single byte (remain %08x)\n",
> + __func__, remainder);
> + num_llis =
> + pl08x_fill_lli_for_desc(pl08x,
> + txd, num_llis, 1,
> + cctl, &remainder);
> + total_bytes++;
> + }
> + }
> + }
> +
> + /*
> + * Send any odd bytes
> + */
> + if (remainder < 0) {
> + dev_err(&pl08x->adev->dev, "%s remainder not fitted 0x%08x bytes\n",
> + __func__, remainder);
> + return 0;
> + }
> +
> + while (remainder) {
> + cctl = pl08x_cctl_bits(cctl, 1, 1, 1);
> + dev_vdbg(&pl08x->adev->dev,
> + "%s align with boundardy, single odd byte (remain %d)\n",
> + __func__, remainder);
> + num_llis = pl08x_fill_lli_for_desc(pl08x, txd, num_llis,
> + 1, cctl, &remainder);
> + total_bytes++;
> + }
> + }
> + if (total_bytes != txd->len) {
> + dev_err(&pl08x->adev->dev,
> + "%s size of encoded lli:s don't match total txd, transferred 0x%08x from size 0x%08x\n",
> + __func__, total_bytes, txd->len);
> + return 0;
> + }
> +
> + if (num_llis >= MAX_NUM_TSFR_LLIS) {
> + dev_err(&pl08x->adev->dev,
> + "%s need to increase MAX_NUM_TSFR_LLIS from 0x%08x\n",
> + __func__, (u32) MAX_NUM_TSFR_LLIS);
> + return 0;
> + }
> + /*
> + * Decide whether this is a loop or a terminated transfer
> + */
> + llis_va = ((struct lli *)txd->llis_va);
> + llis_bus = ((struct lli *)txd->llis_bus);
> +
> + if (cd->circular_buffer) {
> + /*
> + * Loop the circular buffer so that the next element
> + * points back to the beginning of the LLI.
> + */
> + llis_va[num_llis - 1].next =
> + (dma_addr_t)((unsigned int)&(llis_bus[0]) +
> + pl08x->pd->bus_bit_lli);
> + } else {
> + /*
> + * On non-circular buffers, the final LLI terminates
> + * the LLI.
> + */
> + llis_va[num_llis - 1].next = 0;
> + /*
> + * The final LLI element shall also fire an interrupt
> + */
> + llis_va[num_llis - 1].cctl |= PL080_CONTROL_TC_IRQ_EN;
> + }
> +
> + /* Now store the channel register values */
> + txd->csrc = llis_va[0].src;
> + txd->cdst = llis_va[0].dst;
> + if (num_llis > 1)
> + txd->clli = llis_va[0].next;
> + else
> + txd->clli = 0;
> +
> + txd->cctl = llis_va[0].cctl;
> + /* ccfg will be set at physical channel allocation time */
> +
> + {
> + int i;
> +
> + for (i = 0; i < num_llis; i++) {
> + dev_vdbg(&pl08x->adev->dev,
> + "lli %d @%p: csrc=%08x, cdst=%08x, cctl=%08x, clli=%08x\n",
> + i,
> + &llis_va[i],
> + llis_va[i].src,
> + llis_va[i].dst,
> + llis_va[i].cctl,
> + llis_va[i].next
> + );
> + }
> + }
> +
> + /*
> + * Reflects the longest lli submitted so far
> + * TODO: Change to use /proc data
> + */
> + if (pl08x->max_num_llis < num_llis)
> + pl08x->max_num_llis = num_llis;
> +
> + return num_llis;
> +}
> +
> +/* You should call this with the struct pl08x lock held */
> +static void pl08x_free_txd(struct pl08x_driver_data *pl08x, struct pl08x_txd *txd)
> +{
> + if (!txd)
> + dev_err(&pl08x->adev->dev,
> + "%s no descriptor to free\n",
> + __func__);
> +
> + /* Free the LLI */
> + dma_pool_free(pl08x->pool, txd->llis_va,
> + txd->llis_bus);
> +
> + pl08x->pool_ctr--;
> +
> + kfree(txd);
> +}
> +
> +static void pl08x_free_txd_list(struct pl08x_driver_data *pl08x, struct list_head *txdlist)
> +{
> + struct pl08x_txd *txdi = NULL;
> + struct pl08x_txd *next;
> +
> + if (!list_empty(txdlist)) {
> + list_for_each_entry_safe(txdi,
> + next, txdlist, node) {
> + list_del(&txdi->node);
> + pl08x_free_txd(pl08x, txdi);
> + }
> +
> + }
> +}
> +
> +static void pl08x_tasklet(unsigned long data)
> +{
> + struct pl08x_dma_chan *plchan = (struct pl08x_dma_chan *) data;
> + struct pl08x_phy_chan *phychan = plchan->phychan;
> + struct pl08x_driver_data *pl08x = plchan->host;
> + struct pl08x_txd *txdi = NULL;
> + struct pl08x_txd *next;
> + unsigned long flags;
> +
> + if (!plchan)
> + BUG();
> +
> + spin_lock_irqsave(&plchan->lock, flags);
> +
> + if (plchan->at) {
> + dma_async_tx_callback callback =
> + plchan->at->tx.callback;
> + void *callback_param =
> + plchan->at->tx.callback_param;
> +
> + /*
> + * Update last completed
> + */
> + plchan->lc =
> + (plchan->at->tx.cookie);
> +
> + /*
> + * Callback peripheral driver for p/m
> + * to signal completion
> + */
> + if (callback)
> + callback(callback_param);
> +
> + /*
> + * Device callbacks should NOT clear
> + * the current transaction on the channel
> + * Linus: sometimes they should?
> + */
> + if (!plchan->at)
> + BUG();
> +
> + /*
> + * Free the descriptor if it's not for a device
> + * using a circular buffer
> + */
> + if (!plchan->at->cd->circular_buffer) {
> + pl08x_free_txd(pl08x, plchan->at);
> + plchan->at = NULL;
> + }
> + /*
> + * else descriptor for circular
> + * buffers only freed when
> + * client has disabled dma
> + */
> + }
> + /*
> + * If a new descriptor is queued, set it up
> + */
> + if (!list_empty(&plchan->desc_list)) {
> + list_for_each_entry_safe(txdi,
> + next, &plchan->desc_list, node) {
> + list_del_init(&txdi->node);
> + }
> + } else {
> + /*
> + * No more jobs, so free up the physical channel
> + * Free any allocated signal on slave transfers too
> + */
> + if ((phychan->signal >= 0) && pl08x->pd->put_signal)
> + pl08x->pd->put_signal(plchan);
> + pl08x_put_phy_channel(pl08x, phychan);
> + plchan->phychan = NULL;
> + }
> +
> + spin_unlock_irqrestore(&plchan->lock, flags);
> +}
> +
> +static irqreturn_t pl08x_irq(int irq, void *dev)
> +{
> + struct pl08x_driver_data *pl08x = dev;
> + u32 mask = 0;
> + u32 val;
> + int i;
> +
> + val = readl(pl08x->base + PL080_ERR_STATUS);
> + mb();
> + if (val) {
> + /*
> + * An error interrupt (on one or more channels)
> + */
> + dev_err(&pl08x->adev->dev,
> + "%s error interrupt, register value 0x%08x\n",
> + __func__, val);
> + /*
> + * Simply clear ALL PL08X error interrupts,
> + * regardless of channel and cause
> + * FIXME: should be 0x00000003 on PL081 really.
> + */
> + writel(0x000000FF, pl08x->base + PL080_ERR_CLEAR);
> + }
> + val = readl(pl08x->base + PL080_INT_STATUS);
> + mb();
> + for (i = 0; i < pl08x->vd->channels; i++) {
> + if ((1 << i) & val) {
> + /* Locate physical channel */
> + struct pl08x_phy_chan *phychan = &pl08x->phy_chans[i];
> + struct pl08x_dma_chan *plchan = phychan->serving;
> +
> + /* Schedule tasklet on this channel */
> + tasklet_schedule(&plchan->tasklet);
> +
> + mask |= (1 << i);
> + }
> + }
> + /*
> + * Clear only the terminal interrupts on channels we processed
> + */
> + writel(mask, pl08x->base + PL080_TC_CLEAR);
> + mb();
> +
> + return mask ? IRQ_HANDLED : IRQ_NONE;
> +}
> +
> +
> +/*
> + * The DMA ENGINE API
> + */
> +static int pl08x_alloc_chan_resources(struct dma_chan *chan)
> +{
> + return 0;
> +}
> +
> +static void pl08x_free_chan_resources(struct dma_chan *chan)
> +{
> +}
> +
> +/*
> + * This should be called with the channel plchan->lock held
> + */
> +static int prep_phy_channel(struct pl08x_dma_chan *plchan,
> + struct pl08x_txd *txd)
> +{
> + struct pl08x_driver_data *pl08x = plchan->host;
> + struct pl08x_phy_chan *ch;
> + int ret;
> +
> + /* Check if we already have a channel */
> + if (plchan->phychan)
> + return 0;
> +
> + ch = pl08x_get_phy_channel(pl08x, plchan);
> + if (!ch) {
> + /* No physical channel available, cope with it */
> + dev_info(&pl08x->adev->dev, "no physical channel "
> + "available for xfer on %s\n", plchan->name);
> + return -EBUSY;
> + }
> +
> + /*
> + * OK we have a physical channel: for memcpy() this is all we
> + * need, but for slaves the physical siglals may be muxed!
> + * Can the platform allow us to use this channel?
> + */
> + if ((txd->direction == DMA_FROM_DEVICE || txd->direction == DMA_TO_DEVICE) &&
> + pl08x->pd->get_signal) {
> + ret = pl08x->pd->get_signal(plchan);
> + if (ret < 0) {
> + dev_info(&pl08x->adev->dev,
> + "unable to use physical channel "
> + "%d for transfer on %s due to "
> + "platform restrictions\n",
> + ch->id, plchan->name);
> + /* Release physical channel & return */
> + pl08x_put_phy_channel(pl08x, ch);
> + return -EBUSY;
> + }
> + ch->signal = ret;
> + }
> +
> + dev_dbg(&pl08x->adev->dev, "allocated physical "
> + "channel %d and signal %d for xfer on %s\n",
> + ch->id,
> + ch->signal,
> + plchan->name);
> +
> + plchan->phychan = ch;
> +
> + return 0;
> +}
> +
> +/*
> + * First make the LLIs (could/should we do this earlier??)
> + * slave (m/p) - no queued transactions allowed at present
> + * TODO allow queued transactions for non circular buffers
> + * Set up the channel active txd as inactive
> + * m2m - transactions may be queued
> + * If no active txd on channel
> + * set it up as inactive
> + * - issue_pending() will set active & start
> + * else
> + * queue it
> + * Lock channel since there may be (at least for m2m) multiple calls
> + *
> + * Return < 0 for error
> + */
> +
> +static dma_cookie_t pl08x_tx_submit(struct dma_async_tx_descriptor *tx)
> +{
> + int num_llis;
> + unsigned long flags;
> + struct pl08x_txd *txd = container_of(tx, struct pl08x_txd, tx);
> + struct pl08x_dma_chan *plchan = to_pl08x_chan(tx->chan);
> + struct pl08x_driver_data *pl08x = plchan->host;
> + int ret;
> +
> + num_llis = pl08x_fill_llis_for_desc(pl08x, txd);
> +
> + if (num_llis) {
> + spin_lock_irqsave(&plchan->lock, flags);
> + atomic_inc(&plchan->last_issued);
> + tx->cookie = atomic_read(&plchan->last_issued);
> +
> + if (plchan->at) {
> +
> + /*
> + * If this device not using a circular buffer then
> + * queue this new descriptor for transfer.
> + * The descriptor for a circular buffer continues
> + * to be used until the channel is freed.
> + */
> + if (txd->cd->circular_buffer)
> + dev_err(&pl08x->adev->dev,
> + "%s attempting to queue a circular buffer\n",
> + __func__);
> + else
> + list_add_tail(&txd->node,
> + &plchan->desc_list);
> +
> + } else {
> + plchan->at = txd;
> + txd->active = false;
> + }
> +
> + /*
> + * See if we already have a physical channel allocated,
> + * else this is the time to try to get one.
> + */
> + ret = prep_phy_channel(plchan, txd);
> + if (ret) {
> + /* No physical channel available, cope with it */
> + spin_unlock_irqrestore(&plchan->lock, flags);
> + return -EBUSY;
> + }
> +
> + spin_unlock_irqrestore(&plchan->lock, flags);
> +
> + return tx->cookie;
> + } else
> + return -EINVAL;
> +}
> +
> +static struct dma_async_tx_descriptor *pl08x_prep_dma_interrupt(
> + struct dma_chan *chan, unsigned long flags)
> +{
> + struct dma_async_tx_descriptor *retval = NULL;
> +
> + return retval;
> +}
> +
> +/*
> + * Code accessing dma_async_is_complete() in a tight loop
> + * may give problems - could schedule where indicated.
> + * If slaves are relying on interrupts to signal completion this
> + * function must not be called with interrupts disabled
> + */
> +static enum dma_status
> +pl08x_dma_tx_status(struct dma_chan *chan,
> + dma_cookie_t cookie,
> + struct dma_tx_state *txstate)
> +{
> + struct pl08x_dma_chan *plchan = to_pl08x_chan(chan);
> + dma_cookie_t last_used;
> + dma_cookie_t last_complete;
> + enum dma_status ret;
> + u32 bytesleft = 0;
> +
> + last_used = atomic_read(&plchan->last_issued);
> + last_complete = plchan->lc;
> +
> + ret = dma_async_is_complete(cookie, last_complete, last_used);
> + if (ret == DMA_SUCCESS) {
> + dma_set_tx_state(txstate, last_complete, last_used, 0);
> + return ret;
> + }
> +
> + /*
> + * schedule(); could be inserted here
> + */
> +
> + /*
> + * This cookie not complete yet
> + */
> + last_used = atomic_read(&plchan->last_issued);
> + last_complete = plchan->lc;
> +
> + /* Get number of bytes left in the active transaction */
> + if (plchan->phychan)
> + bytesleft = pl08x_getbytes_phy_chan(plchan->phychan);
> +
> + dma_set_tx_state(txstate, last_complete, last_used,
> + bytesleft);
> +
> + return DMA_IN_PROGRESS;
> + /* FIXME: make possible to return DMA_IN_PROGRESS */
> +}
> +
> +/* PrimeCell DMA extension */
> +struct burst_table {
> + int burstwords;
> + u32 reg;
> +};
> +
> +static const struct burst_table burst_sizes[] = {
> + {
> + .burstwords = 256,
> + .reg = (PL080_BSIZE_256 << PL080_CONTROL_SB_SIZE_SHIFT) |
> + (PL080_BSIZE_256 << PL080_CONTROL_DB_SIZE_SHIFT),
> + },
> + {
> + .burstwords = 128,
> + .reg = (PL080_BSIZE_128 << PL080_CONTROL_SB_SIZE_SHIFT) |
> + (PL080_BSIZE_128 << PL080_CONTROL_DB_SIZE_SHIFT),
> + },
> + {
> + .burstwords = 64,
> + .reg = (PL080_BSIZE_64 << PL080_CONTROL_SB_SIZE_SHIFT) |
> + (PL080_BSIZE_64 << PL080_CONTROL_DB_SIZE_SHIFT),
> + },
> + {
> + .burstwords = 32,
> + .reg = (PL080_BSIZE_32 << PL080_CONTROL_SB_SIZE_SHIFT) |
> + (PL080_BSIZE_32 << PL080_CONTROL_DB_SIZE_SHIFT),
> + },
> + {
> + .burstwords = 16,
> + .reg = (PL080_BSIZE_16 << PL080_CONTROL_SB_SIZE_SHIFT) |
> + (PL080_BSIZE_16 << PL080_CONTROL_DB_SIZE_SHIFT),
> + },
> + {
> + .burstwords = 8,
> + .reg = (PL080_BSIZE_8 << PL080_CONTROL_SB_SIZE_SHIFT) |
> + (PL080_BSIZE_8 << PL080_CONTROL_DB_SIZE_SHIFT),
> + },
> + {
> + .burstwords = 4,
> + .reg = (PL080_BSIZE_4 << PL080_CONTROL_SB_SIZE_SHIFT) |
> + (PL080_BSIZE_4 << PL080_CONTROL_DB_SIZE_SHIFT),
> + },
> + {
> + .burstwords = 1,
> + .reg = (PL080_BSIZE_1 << PL080_CONTROL_SB_SIZE_SHIFT) |
> + (PL080_BSIZE_1 << PL080_CONTROL_DB_SIZE_SHIFT),
> + },
> +};
> +
> +static void dma_set_ambaconfig(struct dma_chan *chan,
> + struct amba_dma_channel_config *config)
> +{
> + struct pl08x_dma_chan *plchan = to_pl08x_chan(chan);
> + struct pl08x_driver_data *pl08x = plchan->host;
> + struct pl08x_channel_data *cd = plchan->cd;
> + int maxburst = config->maxburst;
> + u32 cctl = 0;
> + /* Mask out all except src and dst channel */
> + u32 ccfg = cd->ccfg & 0x000003DEU;
> + int i = 0;
> +
> + plchan->amba_addr = config->addr;
> + plchan->amba_direction = config->direction;
> +
> + switch (config->addr_width) {
> + case 1:
> + cctl |= (PL080_WIDTH_8BIT << PL080_CONTROL_SWIDTH_SHIFT) |
> + (PL080_WIDTH_8BIT << PL080_CONTROL_DWIDTH_SHIFT);
> + break;
> + case 2:
> + cctl |= (PL080_WIDTH_16BIT << PL080_CONTROL_SWIDTH_SHIFT) |
> + (PL080_WIDTH_16BIT << PL080_CONTROL_DWIDTH_SHIFT);
> + break;
> + case 4:
> + cctl |= (PL080_WIDTH_32BIT << PL080_CONTROL_SWIDTH_SHIFT) |
> + (PL080_WIDTH_32BIT << PL080_CONTROL_DWIDTH_SHIFT);
> + break;
> + default:
> + dev_err(&pl08x->adev->dev,
> + "bad ambaconfig: alien address width\n");
> + return;
> + }
> +
> + /* Now decide on a maxburst */
> + while (i < ARRAY_SIZE(burst_sizes)) {
> + if (burst_sizes[i].burstwords <= maxburst)
> + break;
> + i++;
> + }
> + cctl |= burst_sizes[i].reg;
> +
> + /* Transfer direction */
> + if (config->direction == DMA_TO_DEVICE) {
> + cctl |= PL080_CONTROL_SRC_INCR;
> + ccfg |= PL080_FLOW_MEM2PER << PL080_CONFIG_FLOW_CONTROL_SHIFT;
> + } else if (config->direction == DMA_FROM_DEVICE) {
> + cctl |= PL080_CONTROL_DST_INCR;
> + ccfg |= PL080_FLOW_PER2MEM << PL080_CONFIG_FLOW_CONTROL_SHIFT;
> + } else {
> + dev_err(&pl08x->adev->dev,
> + "bad ambaconfig: alien transfer direction\n");
> + }
> +
> + /* Access the cell in privileged mode, non-bufferable, non-cacheable */
> + cctl &= ~PL080_CONTROL_PROT_MASK;
> + cctl |= PL080_CONTROL_PROT_SYS;
> +
> + /* Modify the default channel data to fit PrimeCell request */
> + cd->cctl = cctl;
> + cd->ccfg = ccfg;
> +
> + dev_info(&pl08x->adev->dev,
> + "configured channel %s (%s) for %s, data width %d, "
> + "maxburst %d words, LE, CCTL=%08x, CCFG=%08x\n",
> + dma_chan_name(chan), plchan->name,
> + (config->direction == DMA_FROM_DEVICE) ? "RX" : "TX",
> + config->addr_width,
> + config->maxburst,
> + cctl, ccfg);
> +}
> +
> +/*
> + * Slave transactions callback to the slave device to allow
> + * synchronization of slave DMA signals with the DMAC enable
> + */
> +static void pl08x_issue_pending(struct dma_chan *chan)
> +{
> + struct pl08x_dma_chan *plchan = to_pl08x_chan(chan);
> + struct pl08x_driver_data *pl08x = plchan->host;
> + unsigned long flags;
> +
> + spin_lock_irqsave(&plchan->lock, flags);
> + if (plchan->at) {
> + if (!plchan->at->active) {
> + /* Configure the physical channel for the active txd */
> + pl08x_config_phychan_for_txd(plchan);
> + pl08x_set_cregs(pl08x, plchan->phychan);
> + pl08x_enable_phy_chan(pl08x, plchan->phychan);
> + plchan->at->active = true;
> + }
> + /*
> + * else skip active transfer
> + * Calls with active txd occur for NET_DMA
> + * - there can be queued descriptors
> + */
> + }
> + spin_unlock_irqrestore(&plchan->lock, flags);
> + /*
> + * else - calls with no active descriptor occur for NET_DMA
> + */
> +}
> +
> +/*
> + * Initialize a descriptor to be used by memcpy submit
> + */
> +static struct dma_async_tx_descriptor *pl08x_prep_dma_memcpy(
> + struct dma_chan *chan, dma_addr_t dest, dma_addr_t src,
> + size_t len, unsigned long flags)
> +{
> + struct pl08x_dma_chan *plchan = to_pl08x_chan(chan);
> + struct pl08x_driver_data *pl08x = plchan->host;
> + struct pl08x_txd *txd;
> +
> + txd = kzalloc(sizeof(struct pl08x_txd), GFP_KERNEL);
> + if (!txd) {
> + dev_err(&pl08x->adev->dev,
> + "%s no memory for descriptor\n", __func__);
> + return NULL;
> + }
> +
> + dma_async_tx_descriptor_init(&txd->tx, chan);
> + txd->direction = DMA_NONE;
> + txd->srcbus.addr = src;
> + txd->dstbus.addr = dest;
> +
> + /* Set platform data for m2m */
> + txd->cd = &pl08x->pd->memcpy_channel;
> + /* Both to be incremented or the code will break */
> + txd->cd->cctl |= PL080_CONTROL_SRC_INCR | PL080_CONTROL_DST_INCR;
> + txd->tx.tx_submit = pl08x_tx_submit;
> + txd->tx.callback = NULL;
> + txd->tx.callback_param = NULL;
> + txd->len = len;
> +
> + INIT_LIST_HEAD(&txd->node);
> +
> + return &txd->tx;
> +}
> +
> +struct dma_async_tx_descriptor *pl08x_prep_slave_sg(
> + struct dma_chan *chan, struct scatterlist *sgl,
> + unsigned int sg_len, enum dma_data_direction direction,
> + unsigned long flags)
> +{
> + struct pl08x_dma_chan *plchan = to_pl08x_chan(chan);
> + struct pl08x_driver_data *pl08x = plchan->host;
> + struct pl08x_txd *txd;
> +
> + /*
> + * Current implementation ASSUMES only one sg
> + */
> + if (sg_len != 1) {
> + dev_err(&pl08x->adev->dev, "%s prepared too long sglist\n",
> + __func__);
> + BUG();
> + }
> +
> + dev_info(&pl08x->adev->dev, "%s prepare transaction from %s\n",
> + __func__, plchan->name);
> +
> + txd = kmalloc(sizeof(struct pl08x_txd), GFP_KERNEL);
> + if (!txd) {
> + dev_err(&pl08x->adev->dev, "%s no txd\n", __func__);
> + return NULL;
> + }
> +
> + dma_async_tx_descriptor_init(&txd->tx, chan);
> +
> + if (direction != plchan->amba_direction)
> + dev_err(&pl08x->adev->dev, "%s DMA setup does not match "
> + "the direction configured for the PrimeCell\n",
> + __func__);
> +
> + txd->direction = direction;
> + if (direction == DMA_TO_DEVICE) {
> + txd->srcbus.addr = sgl->dma_address;
> + txd->dstbus.addr = plchan->amba_addr;
> + } else if (direction == DMA_FROM_DEVICE) {
> + txd->srcbus.addr = plchan->amba_addr;
> + txd->dstbus.addr = sgl->dma_address;
> + } else {
> + dev_err(&pl08x->adev->dev,
> + "%s direction unsupported\n", __func__);
> + return NULL;
> + }
> + txd->cd = plchan->cd;
> + txd->tx.tx_submit = pl08x_tx_submit;
> + txd->tx.callback = NULL;
> + txd->tx.callback_param = NULL;
> + txd->len = sgl->length;
> +
> + INIT_LIST_HEAD(&txd->node);
> +
> + return &txd->tx;
> +}
> +
> +static int pl08x_control(struct dma_chan *chan, enum dma_ctrl_cmd cmd,
> + unsigned long arg)
> +{
> + struct pl08x_dma_chan *plchan = to_pl08x_chan(chan);
> + struct pl08x_driver_data *pl08x = plchan->host;
> + unsigned long flags;
> +
> + /* Controls applicable to inactive channels */
> + if (cmd == DMA_CONFIG_AMBA) {
> + dma_set_ambaconfig(chan,
> + (struct amba_dma_channel_config *)
> + arg);
> + return 0;
> + }
> +
> + /* Anything succeeds on non-existing transfers */
> + spin_lock_irqsave(&plchan->lock, flags);
> + if (!plchan->at || !plchan->phychan) {
> + spin_unlock_irqrestore(&plchan->lock, flags);
> + return 0;
> + }
> +
> + switch (cmd) {
> + case DMA_TERMINATE_ALL:
> + pl08x_stop_phy_chan(plchan->phychan);
> +
> + /* Mark physical channel as free and free any slave signal */
> + if ((plchan->phychan->signal >= 0) && pl08x->pd->put_signal)
> + pl08x->pd->put_signal(plchan);
> + pl08x_put_phy_channel(pl08x, plchan->phychan);
> + plchan->phychan = NULL;
> +
> + /* Dequeue jobs and free LLIs */
> + pl08x_free_txd(pl08x, plchan->at);
> + pl08x_free_txd_list(pl08x, &plchan->desc_list);
> +
> + spin_unlock_irqrestore(&plchan->lock, flags);
> +
> + return 0;
> + case DMA_PAUSE:
> + pl08x_pause_phy_chan(plchan->phychan);
> + spin_unlock_irqrestore(&plchan->lock, flags);
> + return 0;
> + case DMA_RESUME:
> + pl08x_resume_phy_chan(plchan->phychan);
> + spin_unlock_irqrestore(&plchan->lock, flags);
> + return 0;
> + default:
> + break;
> + }
> + spin_unlock_irqrestore(&plchan->lock, flags);
> +
> + /* Unknown command */
> + return -ENXIO;
> +}
> +
> +bool pl08x_filter_id(struct dma_chan *chan, void *chan_id)
> +{
> + struct pl08x_dma_chan *plchan = to_pl08x_chan(chan);
> + char *name = chan_id;
> +
> + /* Check that the channel is not taken! */
> + if (!strcmp(plchan->name, name))
> + return true;
> +
> + return false;
> +}
> +
> +struct dma_device dmac_memcpy = {
> + .device_alloc_chan_resources = pl08x_alloc_chan_resources,
> + .device_free_chan_resources = pl08x_free_chan_resources,
> + .device_prep_dma_memcpy = pl08x_prep_dma_memcpy,
> + .device_prep_dma_xor = NULL,
> + .device_prep_dma_memset = NULL,
> + .device_prep_dma_interrupt = pl08x_prep_dma_interrupt,
> + .device_tx_status = pl08x_dma_tx_status,
> + .device_issue_pending = pl08x_issue_pending,
> + .device_control = pl08x_control,
> + /*
> + * Align to 4-byte boundary
> + * This makes the DMAtests fail with grace on PB1176
> + * broken DMA hardware instead of locking everything
> + * up.
> + */
> + /* .copy_align = 2, */
> +};
> +
> +struct dma_device dmac_slave = {
they must be marked static.
> + .device_alloc_chan_resources = pl08x_alloc_chan_resources,
> + .device_free_chan_resources = pl08x_free_chan_resources,
> + .device_prep_dma_xor = NULL,
> + .device_prep_dma_memset = NULL,
> + .device_prep_dma_interrupt = pl08x_prep_dma_interrupt,
> + .device_tx_status = pl08x_dma_tx_status,
> + .device_issue_pending = pl08x_issue_pending,
> + .device_prep_slave_sg = pl08x_prep_slave_sg,
> + .device_control = pl08x_control,
> +};
> +
One more thing linus, why do we need to create this separation between
channels. i.e. few are for memcpy and few for slave_sg. Why don't all
channels support everything and this is resolved at runtime.
> +
> +/*
> + * Just check that the device is there and active
> + * TODO: turn this bit on/off depending on the number of
> + * physical channels actually used, if it is zero... well
> + * shut it off.
> + */
> +static void pl08x_ensure_on(struct pl08x_driver_data *pl08x)
> +{
> + u32 val;
> +
> + val = readl(pl08x->base + PL080_CONFIG);
> + val &= ~(PL080_CONFIG_M2_BE | PL080_CONFIG_M1_BE | PL080_CONFIG_ENABLE);
> + /* We implictly clear bit 1 and that means little-endian mode */
> + val |= PL080_CONFIG_ENABLE;
> + mb();
> + writel(val, pl08x->base + PL080_CONFIG);
> + mb();
> +}
> +
> +/*
> + * Initialise the DMAC memcpy channels.
> + * Make a local wrapper to hold required data
> + */
> +static int pl08x_dma_init_memcpy_channels(struct pl08x_driver_data *pl08x,
> + struct dma_device *memdev)
> +{
> + struct pl08x_dma_chan *chan;
> + int i;
> +
> + INIT_LIST_HEAD(&memdev->channels);
> + /*
> + * Register as many many memcpy as we have physical channels,
> + * we won't always be able to use all but the code will have
> + * to cope with that situation.
> + */
> + for (i = 0; i < pl08x->vd->channels; i++) {
> + chan = kzalloc(sizeof(struct pl08x_dma_chan), GFP_KERNEL);
> + if (!chan) {
> + dev_err(&pl08x->adev->dev,
> + "%s no memory for channel\n", __func__);
> + return -ENOMEM;
> + }
> +
> + chan->host = pl08x;
> + chan->name = kasprintf(GFP_KERNEL, "memcpy%d", i);
> + if (!chan->name) {
> + kfree(chan);
> + return -ENOMEM;
> + }
> + chan->cd = &pl08x->pd->memcpy_channel;
> + dev_info(&pl08x->adev->dev,
> + "initialize virtual memcpy channel \"%s\"\n",
> + chan->name);
> +
> + chan->chan.device = memdev;
> + atomic_set(&chan->last_issued, 0);
> + chan->lc = atomic_read(&chan->last_issued);
> +
> + spin_lock_init(&chan->lock);
> + INIT_LIST_HEAD(&chan->desc_list);
> + tasklet_init(&chan->tasklet, pl08x_tasklet,
> + (unsigned long) chan);
> +
> + list_add_tail(&chan->chan.device_node, &memdev->channels);
> + }
> + dev_info(&pl08x->adev->dev, "initialized %d virtual memcpy channels\n", i);
> + return i;
> +}
> +
> +/*
> + * Initialise the DMAC slave channels.
> + * Make a local wrapper to hold required data
> + */
> +static int pl08x_dma_init_slave_channels(struct pl08x_driver_data *pl08x,
> + struct dma_device *slave)
above 2 functions are almost exactly same, can we have single function
instead of two.
> +{
> + struct pl08x_dma_chan *chan;
> + int i;
> +
> + INIT_LIST_HEAD(&slave->channels);
> + for (i = 0; i < pl08x->pd->num_slave_channels; i++) {
> + chan = kzalloc(sizeof(struct pl08x_dma_chan), GFP_KERNEL);
> + if (!chan) {
> + dev_err(&pl08x->adev->dev,
> + "%s no memory for channel\n", __func__);
> + return -ENOMEM;
> + }
> +
> + chan->host = pl08x;
> + chan->name = pl08x->pd->slave_channels[i].bus_id;
> + chan->cd = &pl08x->pd->slave_channels[i];
> + dev_info(&pl08x->adev->dev,
> + "initialize virtual channel \"%s\"\n",
> + chan->name);
> +
> + chan->chan.device = slave;
> + atomic_set(&chan->last_issued, 0);
> + chan->lc = atomic_read(&chan->last_issued);
> +
> + spin_lock_init(&chan->lock);
> + INIT_LIST_HEAD(&chan->desc_list);
> + tasklet_init(&chan->tasklet, pl08x_tasklet,
> + (unsigned long) chan);
> +
> + list_add_tail(&chan->chan.device_node, &slave->channels);
> + }
> + dev_info(&pl08x->adev->dev, "initialized %d virtual slave channels\n", i);
> + return i;
> +}
> +
> +static int pl08x_probe(struct amba_device *adev, struct amba_id *id)
> +{
> + struct pl08x_driver_data *pl08x;
> + struct vendor_data *vd = id->data;
> + int ret = 0;
> + int i;
> +
> + ret = amba_request_regions(adev, NULL);
> + if (ret)
> + return ret;
> +
> + /* Create the driver state holder */
> + pl08x = kzalloc(sizeof(struct pl08x_driver_data), GFP_KERNEL);
> + if (!pl08x) {
> + ret = -ENOMEM;
> + goto out_no_pl08x;
> + }
> +
> + /* Assign useful pointers to the driver state */
> + pl08x->adev = adev;
> + pl08x->vd = vd;
> +
> + /* A DMA memory pool for LLIs, align on 1-byte boundary */
> + pl08x->pool = dma_pool_create(DRIVER_NAME, &pl08x->adev->dev,
> + PL08X_LLI_TSFR_SIZE, PL08X_ALIGN, 0);
> + if (!pl08x->pool) {
> + ret = -ENOMEM;
> + goto out_no_lli_pool;
> + }
> + pl08x->pool_ctr = 0;
> + pl08x->max_num_llis = 0;
they are already 0.
> +
> + spin_lock_init(&pl08x->lock);
> +
> + pl08x->base = ioremap(adev->res.start, resource_size(&adev->res));
> + if (!pl08x->base) {
> + ret = -ENOMEM;
> + goto out_no_ioremap;
> + }
> +
> + /* Turn on the PL08x */
> + pl08x_ensure_on(pl08x);
> +
> + /*
> + * Attach the interrupt handler
> + */
> + writel(0x000000FF, pl08x->base + PL080_ERR_CLEAR);
> + writel(0x000000FF, pl08x->base + PL080_TC_CLEAR);
> + mb();
> +
> + ret = request_irq(adev->irq[0], pl08x_irq, IRQF_DISABLED,
> + vd->name, pl08x);
> + if (ret) {
> + dev_err(&adev->dev, "%s failed to request "
> + "interrupt %d\n",
> + __func__, adev->irq[0]);
can be written in 2 lines only.
> + goto out_no_irq;
> + }
> +
> + /* Initialize physical channels */
> + pl08x->phy_chans = kmalloc((vd->channels * sizeof(struct pl08x_phy_chan)),
> + GFP_KERNEL);
> + if (!pl08x->phy_chans) {
> + dev_err(&adev->dev, "%s failed to allocate "
> + "physical channel holders\n",
> + __func__);
> + goto out_no_phychans;
> + }
> +
> + for (i = 0; i < vd->channels; i++) {
> + struct pl08x_phy_chan *ch = &pl08x->phy_chans[i];
> +
> + ch->id = i;
> + ch->base = pl08x->base + PL080_Cx_BASE(i);
> + spin_lock_init(&ch->lock);
> + ch->serving = NULL;
> + ch->signal = -1;
> + dev_info(&adev->dev,
> + "physical channel %d is %s\n", i,
> + pl08x_phy_channel_busy(ch) ? "BUSY" : "FREE");
> + }
> +
> + /* Get the platform data */
> + pl08x->pd = (struct pl08x_platform_data *)(adev->dev.platform_data);
better to use dev_get_platdata, also no need of typecasting as
platform_data is of type void*.
> +
> + /* Set caps */
> + dma_cap_set(DMA_MEMCPY, dmac_memcpy.cap_mask);
> + dma_cap_set(DMA_SLAVE, dmac_slave.cap_mask);
> + dmac_memcpy.dev = &adev->dev;
> + dmac_slave.dev = &adev->dev;
> +
> + /* Register memcpy channels */
> + ret = pl08x_dma_init_memcpy_channels(pl08x, &dmac_memcpy);
> + if (ret <= 0) {
> + dev_warn(&pl08x->adev->dev,
> + "%s failed to enumerate memcpy channels - %d\n",
> + __func__, ret);
> + goto out_no_memcpy;
> + }
> + dmac_memcpy.chancnt = ret;
> +
> + /* Register slave channels */
> + ret = pl08x_dma_init_slave_channels(pl08x, &dmac_slave);
> + if (ret <= 0) {
> + dev_warn(&pl08x->adev->dev,
> + "%s failed to enumerate slave channels - %d\n",
> + __func__, ret);
> + goto out_no_slave;
> + }
> + dmac_slave.chancnt = ret;
> +
> + ret = dma_async_device_register(&dmac_memcpy);
> + if (ret) {
> + dev_warn(&pl08x->adev->dev,
> + "%s failed to register memcpy as an async device - %d\n",
> + __func__, ret);
> + goto out_no_memcpy_reg;
> + }
> +
> + ret = dma_async_device_register(&dmac_slave);
> + if (ret) {
> + dev_warn(&pl08x->adev->dev,
> + "%s failed to register slave as an async device - %d\n",
> + __func__, ret);
> + goto out_no_slave_reg;
> + }
> +
> + amba_set_drvdata(adev, pl08x);
> + dev_info(&pl08x->adev->dev, "ARM(R) %s DMA block initialized @%08x\n",
> + vd->name, adev->res.start);
> + return 0;
> +
> +out_no_slave_reg:
> + dma_async_device_unregister(&dmac_memcpy);
> +out_no_memcpy_reg:
> + /* FIXME: free slave channels */
> +out_no_slave:
> + /* FIXME: free memcpy channels */
> +out_no_memcpy:
> + kfree(pl08x->phy_chans);
> +out_no_phychans:
> + free_irq(adev->irq[0], pl08x);
> +out_no_irq:
> + iounmap(pl08x->base);
> +out_no_ioremap:
> + dma_pool_destroy(pl08x->pool);
> +out_no_lli_pool:
> + kfree(pl08x);
> +out_no_pl08x:
> + amba_release_regions(adev);
> + return ret;
> +}
> +
> +/* PL080 has 8 channels and the PL080 have just 2 */
"PL080 or PL081 have just 2"?? You mentioned at the beginning of this
file, that PL081 has 16 channels. Am i missing something??
> +static struct vendor_data vendor_pl080 = {
> + .name = "PL080",
> + .channels = 8,
> +};
> +
> +static struct vendor_data vendor_pl081 = {
> + .name = "PL081",
> + .channels = 2,
> +};
> +
> +static struct amba_id pl08x_ids[] = {
> + /* PL080 */
> + {
> + .id = 0x00041080,
> + .mask = 0x000fffff,
> + .data = &vendor_pl080,
> + },
> + /* PL081 */
> + {
> + .id = 0x00041081,
> + .mask = 0x000fffff,
> + .data = &vendor_pl081,
> + },
> + /* Nomadik 8815 PL080 variant */
> + {
> + .id = 0x00280880,
> + .mask = 0x00ffffff,
> + .data = &vendor_pl080,
> + },
> + { 0, 0 },
> +};
> +
> +static struct amba_driver pl08x_amba_driver = {
> + .drv.name = DRIVER_NAME,
> + .id_table = pl08x_ids,
> + .probe = pl08x_probe,
> +};
> +
> +static int __init pl08x_init(void)
> +{
> + int retval;
> + retval = amba_driver_register(&pl08x_amba_driver);
> + if (retval)
> + printk(KERN_WARNING
> + "PL08X::pl08x_init() - failed to register as an amba device - %d\n",
> + retval);
> + return retval;
> +}
> +subsys_initcall(pl08x_init);
> diff --git a/include/linux/amba/pl08x.h b/include/linux/amba/pl08x.h
> new file mode 100644
> index 0000000..5f9b16f
> --- /dev/null
> +++ b/include/linux/amba/pl08x.h
> @@ -0,0 +1,173 @@
> +/*
> + * linux/amba/pl08x.h - ARM PrimeCell DMA Controller driver
> + *
> + * Copyright (C) 2005 ARM Ltd
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * pl08x information required by platform code
> + *
> + * Please credit ARM.com
> + * Documentation: ARM DDI 0196D
> + *
> + */
> +
> +#ifndef AMBA_PL08X_H
> +#define AMBA_PL08X_H
> +
> +/* We need sizes of structs from this header */
> +#include <linux/dmaengine.h>
> +
> +/**
> + * struct pl08x_channel_data - data structure to pass info between
> + * platform and PL08x driver regarding channel configuration
> + * @bus_id: name of this device channel, not just a device name since
> + * devices may have more than one channel e.g. "foo_tx"
> + * @min_signal: the minimum DMA signal number to be muxed in for this
> + * channel (for platforms supporting muxed signals). If you have
> + * static assignments, make sure this is set to the assigned signal
> + * number, PL08x have 16 possible signals in number 0 thru 15 so
> + * when these are not enough they often get muxed (in hardware)
> + * disabling simultaneous use of the same channel for two devices.
> + * @max_signal: the maximum DMA signal number to be muxed in for
> + * the channel. Set to the same as min_signal for
> + * devices with static assignments
> + * @muxval: a number usually used to poke into some mux regiser to
> + * mux in the signal to this channel
> + * @cctl_opt: default options for the channel control register
> + * @circular_buffer: whether the buffer passed in is circular and
> + * shall simply be looped round round (like a record baby round
> + * round round round)
> + */
> +struct pl08x_channel_data {
> + char *bus_id;
> + int min_signal;
> + int max_signal;
> + u32 muxval;
> + unsigned int cctl; /* Turn me into u32? */
> + u32 ccfg;
> + bool circular_buffer;
> +};
> +
> +/**
> + * struct pl08x_bus_data - information of source or destination
> + * busses for a transfer
> + * @addr: current address
> + * @maxwidth: the maximum width of a transfer on this bus
> + * @buswidth: the width of this bus in bytes: 1, 2 or 4
> + * @fill_bytes: bytes required to fill to the next bus memory
> + * boundary
> + */
> +struct pl08x_bus_data {
> + dma_addr_t addr;
> + u8 maxwidth;
> + u8 buswidth;
> + u32 fill_bytes;
> +};
> +
> +/**
> + * struct pl08x_phy_chan - holder for the physical channels
> + * @id: physical index to this channel
> + * @lock: a lock to use when altering an instance of this struct
> + * @signal: the physical signal (aka channel) serving this
> + * physical channel right now
> + * @serving: the virtual channel currently being served by this
> + * physical channel
> + */
> +struct pl08x_phy_chan {
> + unsigned int id;
> + void __iomem *base;
> + spinlock_t lock;
> + int signal;
> + struct pl08x_dma_chan *serving;
> + u32 csrc;
> + u32 cdst;
> + u32 clli;
> + u32 cctl;
> + u32 ccfg;
> +};
> +
> +/**
> + * struct pl08x_txd - wrapper for struct dma_async_tx_descriptor
> + * @llis_bus: DMA memory address (physical) start for the LLIs
> + * @llis_va: virtual memory address start for the LLIs
> + */
> +struct pl08x_txd {
> + struct dma_async_tx_descriptor tx;
> + enum dma_data_direction direction;
> + struct pl08x_bus_data srcbus;
> + struct pl08x_bus_data dstbus;
> + int len;
> + dma_addr_t llis_bus;
> + void *llis_va;
> + struct list_head node;
> + struct pl08x_channel_data *cd;
> + bool active;
> + /* Settings to be put into the physical channel when we submit this txd */
> + u32 csrc;
> + u32 cdst;
> + u32 clli;
> + u32 cctl;
> +};
> +
> +/**
> + * struct pl08x_dma_chan - this structure wraps a DMA ENGINE channel
> + * @chan: wrappped abstract channel
> + * @phychan: the physical channel utilized by this channel, if there is one
> + * @tasklet: tasklet scheduled by the IRQ to handle actual work etc
> + * @name: name of channel
> + * @cd: channel platform data
> + * @amba_addr: address for RX/TX according to the PrimeCell config
> + * @amba_direction: current direction of this channel according to
> + * @lc: last completed transaction on this channel
> + * @desc_list: queued transactions pending on this channel
> + * @at: active transaction on this channel
> + * @lock: a lock for this channel data
> + * @host: a pointer to the host (internal use)
> + */
> +struct pl08x_dma_chan {
> + struct dma_chan chan;
> + struct pl08x_phy_chan *phychan;
> + struct tasklet_struct tasklet;
> + char *name;
> + struct pl08x_channel_data *cd;
> + dma_addr_t amba_addr;
> + enum dma_data_direction amba_direction;
> + atomic_t last_issued;
> + dma_cookie_t lc;
> + struct list_head desc_list;
> + struct pl08x_txd *at;
> + spinlock_t lock;
> + void *host;
> +};
> +
> +/**
> + * struct pl08x_platform_data - the platform configuration for the
> + * PL08x PrimeCells.
> + * @slave_channels: the channels defined for the different devices on the
> + * platform, all inclusive, including multiplexed channels. The available
> + * physical channels will be multiplexed around these signals as they
> + * are requested, just enumerate all possible channels.
> + * @get_signal: request a physical signal to be used for a DMA
> + * transfer immediately: if there is some multiplexing or similar blocking
> + * the use of the channel the transfer can be denied by returning
> + * less than zero, else it returns the allocated signal number
> + * @put_signal: indicate to the platform that this physical signal is not
> + * running any DMA transfer and multiplexing can be recycled
> + * @bus_bit_lli: Bit[0] of the address indicated which AHB bus master the
> + * LLI addresses are on 0/1 Master 1/2.
> + */
> +struct pl08x_platform_data {
> + struct pl08x_channel_data *slave_channels;
> + unsigned int num_slave_channels;
> + struct pl08x_channel_data memcpy_channel;
> + int (*get_signal)(struct pl08x_dma_chan *);
> + void (*put_signal)(struct pl08x_dma_chan *);
> + unsigned int bus_bit_lli:1;
> +};
> +
> +bool pl08x_filter_id(struct dma_chan *chan, void *chan_id);
> +
> +#endif /* AMBA_PL08X_H */
Hi Viresh, thanks a lot for reviewing this and I'd be *very* happy if
you could give it a spin on
the SPEAr as well!
2010/6/14 Viresh KUMAR <[email protected]>:
>> diff --git a/drivers/dma/amba-pl08x.c b/drivers/dma/amba-pl08x.c
>> (...)
>> + * For peripherals with a FIFO:
>> + * Source ? ? ?burst size == half the depth of the peripheral FIFO
>> + * Destination burst size == width of the peripheral FIFO
>> + *
>
> I didn't get it completely, why burst depends upon width of peripheral FIFO.
I think this is just the wrong word, it should be "depth".
>> + * (Bursts are irrelevant for mem to mem transfers - there are no burst
>> + * signals)
>
> I agree that there are no request lines from memories but still we can program
> them with burst in order to faster the transfer. This burst feature is
> automatically handled by DMA.
Actually in the example platform data I set this to the maxburst size
256 Bytes, however if I read the manual correctly this is simply ignored
when you do mem2mem transfers.
It will simply AHB master the bus until the transaction is finished. That
is why the manual states:
"You must program memory-to-memory transfers with a low channel
priority, otherwise:
? other DMA channels cannot access the bus until the
memory-to-memory transfer
has finished
? other AHB masters cannot perform any transaction."
>> + * @max_num_llis: maximum number of LLIs, i.e. longest linked transfer
>> + * length, submitted so far
>
> What is the significance of this field? What it is used for?
Statistics in Peters original implementation. I'll remove it.
>> +static void pl08x_set_cregs(struct pl08x_driver_data *pl08x,
>> + ? ? ? ? ? ? ? ? ? ? ? ? struct pl08x_phy_chan *ch)
>> +{
>> + ? ? u32 val;
>> +
>> + ? ? /* Wait for channel inactive */
>> + ? ? val = readl(ch->base + PL080_CH_CONFIG);
>> + ? ? while (val & PL080_CONFIG_ACTIVE)
>> + ? ? ? ? ? ? val = readl(ch->base + PL080_CH_CONFIG);
>
> can we use pl08x_phy_channel_busy() instead of above code?
Fixed.
>> + ? ? /*
>> + ? ? ?* Do not access config register until channel shows as inactive
>> + ? ? ?*/
>> + ? ? val = readl(ch->base + PL080_CH_CONFIG);
>> + ? ? while ((val & PL080_CONFIG_ACTIVE) || (val & PL080_CONFIG_ENABLE))
>> + ? ? ? ? ? ? val = readl(ch->base + PL080_CH_CONFIG);
>
> above 3 fns are always called in order, i.e. pl08x_enable_phy_chan will
> be called after pl08x_set_cregs, so we may not require these checks
> here. Is my understanding correct??
The previous check if the channel is active before proceeding, this check also
checks the enable bit, this behaviour comes straight from the manual and is
required to avoid hardware races, so I don't dare to touch it really...
>> + ? ? /* Wait for channel inactive */
>> + ? ? val = readl(ch->base + PL080_CH_CONFIG);
>> + ? ? while (val & PL080_CONFIG_ACTIVE)
>> + ? ? ? ? ? ? val = readl(ch->base + PL080_CH_CONFIG);
>
> can we use pl08x_phy_channel_busy() instead of above code?
> Please check everywhere.
It's just these two places. Fixed this one as well.
>> +
>> + ? ? mb();
>> +
>> + ? ? return;
>
> return not required!!
Fixed.
>> +static void pl08x_resume_phy_chan(struct pl08x_phy_chan *ch)
>
> can these small fns be made inline???
Probably but the compiler will do that anyway if there some
point. I'm afraid of violating chapter 15 of CodingStyle...
>> +/* Stops the channel */
>> +static void pl08x_stop_phy_chan(struct pl08x_phy_chan *ch)
>> +{
>> + ? ? u32 val;
>> +
>> + ? ? pl08x_pause_phy_chan(ch);
>> +
>> + ? ? /* Disable channel */
>> + ? ? val = readl(ch->base + PL080_CH_CONFIG);
>> + ? ? val &= ~PL080_CONFIG_ENABLE;
>> + ? ? writel(val, ch->base + PL080_CH_CONFIG);
>> + ? ? mb();
>> +
>> + ? ? return;
>
> same here. return not required.
Fixed all these.
>> +static inline u32 get_bytes_in_cctl(u32 cctl)
>> +{
>> + ? ? /* The source width defines the number of bytes */
>> + ? ? u32 bytes = cctl & PL080_CONTROL_TRANSFER_SIZE_MASK;
>> +
>> + ? ? switch ((cctl >> 18) & 3) {
>
> better to use Macros instead of magic numbers here!!!
Fixed.
>> +static inline u32 pl08x_cctl_bits(u32 cctl,
>> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? u8 srcwidth,
>> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? u8 dstwidth,
>> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? u32 tsize)
>
> Not sure, if we should write above function prototype in just 2 lines,
> or is it okay to write it the way it is written.
Whatever, I made it two lines instead.
>> +struct dma_device dmac_slave = {
>
> they must be marked static.
Fixed.
>> + ? ? .device_alloc_chan_resources ? ?= pl08x_alloc_chan_resources,
>> + ? ? .device_free_chan_resources ? ? = pl08x_free_chan_resources,
>> + ? ? .device_prep_dma_xor ? ? ? ? ? ?= NULL,
>> + ? ? .device_prep_dma_memset ? ? ? ? = NULL,
>> + ? ? .device_prep_dma_interrupt ? ? ?= pl08x_prep_dma_interrupt,
>> + ? ? .device_tx_status ? ? ? ? ? ? ? = pl08x_dma_tx_status,
>> + ? ? .device_issue_pending ? ? ? ? ? = pl08x_issue_pending,
>> + ? ? .device_prep_slave_sg ? ? ? ? ? = pl08x_prep_slave_sg,
>> + ? ? .device_control ? ? ? ? ? ? ? ? = pl08x_control,
>> +};
>> +
>
> One more thing linus, why do we need to create this separation between
> channels. i.e. few are for memcpy and few for slave_sg. Why don't all
> channels support everything and this is resolved at runtime.
This is done in all in-tree drivers, the reason is (I think) that for example
the dmatest.c test client will look for:
if (dma_has_cap(DMA_MEMCPY, dma_dev->cap_mask)) {
cnt = dmatest_add_threads(dtc, DMA_MEMCPY);
thread_count += cnt > 0 ? cnt : 0;
So if you want to partition some channels for device I/O (typically some
which are hard-coded to some devices) and others for memcpy() you create
a slave instance for the former and a memcpy() instance for the latter.
In this case we multiplex the memcpy and slave transfers on the few
physical channels we have, but I haven't finally decided how to handle this:
perhaps we should always set on physical channel aside for memcpy
so this won't ever fail, and then this special memcpy device entry will help.
Ideas? Use cases?
>> +/*
>> + * Initialise the DMAC slave channels.
>> + * Make a local wrapper to hold required data
>> + */
>> +static int pl08x_dma_init_slave_channels(struct pl08x_driver_data *pl08x,
>> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? struct dma_device *slave)
>
> above 2 functions are almost exactly same, can we have single function
> instead of two.
OK Fixed.
>> + ? ? /* A DMA memory pool for LLIs, align on 1-byte boundary */
>> + ? ? pl08x->pool = dma_pool_create(DRIVER_NAME, &pl08x->adev->dev,
>> + ? ? ? ? ? ? ? ? ? ? PL08X_LLI_TSFR_SIZE, PL08X_ALIGN, 0);
>> + ? ? if (!pl08x->pool) {
>> + ? ? ? ? ? ? ret = -ENOMEM;
>> + ? ? ? ? ? ? goto out_no_lli_pool;
>> + ? ? }
>> + ? ? pl08x->pool_ctr = 0;
>> + ? ? pl08x->max_num_llis = 0;
>
> they are already 0.
OK deleted.
>> + ? ? ret = request_irq(adev->irq[0], pl08x_irq, IRQF_DISABLED,
>> + ? ? ? ? ? ? ? ? ? ? ? vd->name, pl08x);
>> + ? ? if (ret) {
>> + ? ? ? ? ? ? dev_err(&adev->dev, "%s failed to request "
>> + ? ? ? ? ? ? ? ? ? ? "interrupt %d\n",
>> + ? ? ? ? ? ? ? ? ? ? __func__, adev->irq[0]);
>
> can be written in 2 lines only.
Fixed.
>> + ? ? /* Get the platform data */
>> + ? ? pl08x->pd = (struct pl08x_platform_data *)(adev->dev.platform_data);
>
> better to use dev_get_platdata, also no need of typecasting as
> platform_data is of type void*.
Fixed.
>> +/* PL080 has 8 channels and the PL080 have just 2 */
>
> "PL080 or PL081 have just 2"?? You mentioned at the beginning of this
> file, that PL081 has 16 channels. Am i missing something??
Wrong words again, this is one of the most confusing things about PL080/PL081,
it has 2 or 8 *channels* but always 16 *signals*.
Almost all other interrupt controllers have a 1-to-1 correspondence between the
number of incoming signals and the number of available slave channels.
Not the PL08x... it has less channels than signals.
I will underscore it again... if it confuses both me and you it will invariably
confuse everybody else too.
Thanks!
Yours,
Linus Walleij
On 6/14/2010 7:09 PM, Linus Walleij wrote:
> Hi Viresh, thanks a lot for reviewing this and I'd be *very* happy if
> you could give it a spin on
> the SPEAr as well!
I would be happy too linus, will do it in few weeks, right now we are running
short of time.
>
> 2010/6/14 Viresh KUMAR <[email protected]>:
>>> diff --git a/drivers/dma/amba-pl08x.c b/drivers/dma/amba-pl08x.c
>>> (...)
>>> + * (Bursts are irrelevant for mem to mem transfers - there are no burst
>>> + * signals)
>>
>> I agree that there are no request lines from memories but still we can program
>> them with burst in order to faster the transfer. This burst feature is
>> automatically handled by DMA.
>
> Actually in the example platform data I set this to the maxburst size
> 256 Bytes, however if I read the manual correctly this is simply ignored
> when you do mem2mem transfers.
>
> It will simply AHB master the bus until the transaction is finished. That
> is why the manual states:
>
> "You must program memory-to-memory transfers with a low channel
> priority, otherwise:
> ? other DMA channels cannot access the bus until the
> memory-to-memory transfer
> has finished
> ? other AHB masters cannot perform any transaction."
Yes i have seen them. One more thing i have found related to mem to mem
transfers is:
"You must set this value to the burst size of the destination peripheral,
or if the destination is memory, to the memory boundary size."
>>> +static void pl08x_set_cregs(struct pl08x_driver_data *pl08x,
>>> + struct pl08x_phy_chan *ch)
>>> +{
>>> + u32 val;
>>> +
>>> + /* Wait for channel inactive */
>>> + val = readl(ch->base + PL080_CH_CONFIG);
>>> + while (val & PL080_CONFIG_ACTIVE)
>>> + val = readl(ch->base + PL080_CH_CONFIG);
>>
>> can we use pl08x_phy_channel_busy() instead of above code?
>
> Fixed.
>
>>> + /*
>>> + * Do not access config register until channel shows as inactive
>>> + */
>>> + val = readl(ch->base + PL080_CH_CONFIG);
>>> + while ((val & PL080_CONFIG_ACTIVE) || (val & PL080_CONFIG_ENABLE))
>>> + val = readl(ch->base + PL080_CH_CONFIG);
>>
>> above 3 fns are always called in order, i.e. pl08x_enable_phy_chan will
>> be called after pl08x_set_cregs, so we may not require these checks
>> here. Is my understanding correct??
>
> The previous check if the channel is active before proceeding, this check also
> checks the enable bit, this behaviour comes straight from the manual and is
> required to avoid hardware races, so I don't dare to touch it really...
>
I was talking about the similar check present in pl08x_set_cregs, which will
be called before this routine. pl08x_set_cregs has already checked if channel
is active or not. so checking active in this routine is not required.
>>> + .device_alloc_chan_resources = pl08x_alloc_chan_resources,
>>> + .device_free_chan_resources = pl08x_free_chan_resources,
>>> + .device_prep_dma_xor = NULL,
>>> + .device_prep_dma_memset = NULL,
>>> + .device_prep_dma_interrupt = pl08x_prep_dma_interrupt,
>>> + .device_tx_status = pl08x_dma_tx_status,
>>> + .device_issue_pending = pl08x_issue_pending,
>>> + .device_prep_slave_sg = pl08x_prep_slave_sg,
>>> + .device_control = pl08x_control,
>>> +};
>>> +
>>
>> One more thing linus, why do we need to create this separation between
>> channels. i.e. few are for memcpy and few for slave_sg. Why don't all
>> channels support everything and this is resolved at runtime.
>
> This is done in all in-tree drivers, the reason is (I think) that for example
> the dmatest.c test client will look for:
>
> if (dma_has_cap(DMA_MEMCPY, dma_dev->cap_mask)) {
> cnt = dmatest_add_threads(dtc, DMA_MEMCPY);
> thread_count += cnt > 0 ? cnt : 0;
>
> So if you want to partition some channels for device I/O (typically some
> which are hard-coded to some devices) and others for memcpy() you create
> a slave instance for the former and a memcpy() instance for the latter.
>
> In this case we multiplex the memcpy and slave transfers on the few
> physical channels we have, but I haven't finally decided how to handle this:
> perhaps we should always set on physical channel aside for memcpy
> so this won't ever fail, and then this special memcpy device entry will help.
>
> Ideas? Use cases?
Hmmm. I am not sure, but i think we can't hard code a channel for some device.
All channels should be available with both capabilities. If still there are
some conditions (that you might know), where we need to hard code channels
for devices, then this should come from plat data in some way.
I have few more doubts that i wanted to ask. Are following supported in your
driver, we need them in SPEAr:
- Configure burst size of source or destination.
- Configure DMA Master for src or dest.
- Transfer from Peripheral to Peripheral.
- Configure Width of src or dest peripheral.
- Configure Flow controller of transfer.
- Some callback for fixing Request line multiplexing just before
initiating transfer.
- Multiple sg elements in slave_sg transfer. I think it is not supported.
- Control for autoincrement of addresses, both in case of memory and
peripherals.
viresh.
Linus Walleij wrote:
>
> This creates a DMAengine driver for the ARM PL080/PL081 PrimeCells
> based on the implementation earlier submitted by Peter Pearse.
> This is working like a charm for memcpy on the PB11MPCore, but
> slave DMA to devices is still not working.
>
> This DMA controller is used in mostly unmodified form in the ARM
> RealView and Versatile platforms, in the ST-Ericsson Nomadik, and
> in the ST SPEAr platform.
>
> It has been converted to use the header from the Samsung PL080
> derivate instead of its own defintions, and can potentially support
> several controllers in the same system.
>
> Cc: Peter Pearse <[email protected]>
> Cc: Ben Dooks <[email protected]>
> Cc: Kukjin Kim <[email protected]>
Looks good, but please give me some time to test on the board(SMDK6410).
If any problem, let you know. Of course no problem, will ack.
> Cc: Alessandro Rubini <[email protected]>
> Cc: Viresh Kumar <[email protected]>
> Signed-off-by: Linus Walleij <[email protected]>
> ---
> arch/arm/include/asm/hardware/pl080.h | 2 +
> drivers/dma/Kconfig | 9 +
> drivers/dma/Makefile | 1 +
> drivers/dma/amba-pl08x.c | 1925
> +++++++++++++++++++++++++++++++++
> include/linux/amba/pl08x.h | 173 +++
> 5 files changed, 2110 insertions(+), 0 deletions(-)
> create mode 100644 drivers/dma/amba-pl08x.c
> create mode 100644 include/linux/amba/pl08x.h
>
(snip)
Thanks.
Best regards,
Kgene.
--
Kukjin Kim <[email protected]>, Senior Engineer,
SW Solution Development Team, Samsung Electronics Co., Ltd.
On Tue, Jun 15, 2010 at 7:25 PM, Kukjin Kim <[email protected]> wrote:
> Linus Walleij wrote:
>>
>> This creates a DMAengine driver for the ARM PL080/PL081 PrimeCells
>> based on the implementation earlier submitted by Peter Pearse.
>> This is working like a charm for memcpy on the PB11MPCore, but
>> slave DMA to devices is still not working.
>>
>> This DMA controller is used in mostly unmodified form in the ARM
>> RealView and Versatile platforms, in the ST-Ericsson Nomadik, and
>> in the ST SPEAr platform.
>>
>> It has been converted to use the header from the Samsung PL080
>> derivate instead of its own defintions, and can potentially support
>> several controllers in the same system.
>>
>> Cc: Peter Pearse <[email protected]>
>> Cc: Ben Dooks <[email protected]>
>> Cc: Kukjin Kim <[email protected]>
>
> Looks good, but please give me some time to test on the board(SMDK6410).
> If any problem, let you know. Of course no problem, will ack.
Samsung doesn't use the DMA API, so this driver is unlikely to work.
Op 15-06-10 12:45, Jassi Brar schreef:
> On Tue, Jun 15, 2010 at 7:25 PM, Kukjin Kim <[email protected]> wrote:
>> Linus Walleij wrote:
>>>
>>> This creates a DMAengine driver for the ARM PL080/PL081 PrimeCells
>>> based on the implementation earlier submitted by Peter Pearse.
>>> This is working like a charm for memcpy on the PB11MPCore, but
>>> slave DMA to devices is still not working.
>>>
>>> This DMA controller is used in mostly unmodified form in the ARM
>>> RealView and Versatile platforms, in the ST-Ericsson Nomadik, and
>>> in the ST SPEAr platform.
>>>
>>> It has been converted to use the header from the Samsung PL080
>>> derivate instead of its own defintions, and can potentially support
>>> several controllers in the same system.
>>>
>>> Cc: Peter Pearse <[email protected]>
>>> Cc: Ben Dooks <[email protected]>
>>> Cc: Kukjin Kim <[email protected]>
>>
>> Looks good, but please give me some time to test on the board(SMDK6410).
>> If any problem, let you know. Of course no problem, will ack.
> Samsung doesn't use the DMA API, so this driver is unlikely to work.
It doesn't indeed, but it could be adapted to be a wrapper around the DMA engine API.
Or even better, the drivers could be adapted to use that API.
On Tue, Jun 15, 2010 at 8:17 PM, Maurus Cuelenaere
<[email protected]> wrote:
> Op 15-06-10 12:45, Jassi Brar schreef:
>> On Tue, Jun 15, 2010 at 7:25 PM, Kukjin Kim <[email protected]> wrote:
>>> Linus Walleij wrote:
>>>>
>>>> This creates a DMAengine driver for the ARM PL080/PL081 PrimeCells
>>>> based on the implementation earlier submitted by Peter Pearse.
>>>> This is working like a charm for memcpy on the PB11MPCore, but
>>>> slave DMA to devices is still not working.
>>>>
>>>> This DMA controller is used in mostly unmodified form in the ARM
>>>> RealView and Versatile platforms, in the ST-Ericsson Nomadik, and
>>>> in the ST SPEAr platform.
>>>>
>>>> It has been converted to use the header from the Samsung PL080
>>>> derivate instead of its own defintions, and can potentially support
>>>> several controllers in the same system.
>>>>
>>>> Cc: Peter Pearse <[email protected]>
>>>> Cc: Ben Dooks <[email protected]>
>>>> Cc: Kukjin Kim <[email protected]>
>>>
>>> Looks good, but please give me some time to test on the board(SMDK6410).
>>> If any problem, let you know. Of course no problem, will ack.
>> Samsung doesn't use the DMA API, so this driver is unlikely to work.
>
> It doesn't indeed, but it could be adapted to be a wrapper around the DMA engine API.
> Or even better, the drivers could be adapted to use that API.
I don't particularly like the idea of making Samsung's drivers use the DMA API.
IMHO the S3C dma api is better atm.
Op 15-06-10 13:39, Jassi Brar schreef:
> On Tue, Jun 15, 2010 at 8:17 PM, Maurus Cuelenaere
> <[email protected]> wrote:
>
>> Op 15-06-10 12:45, Jassi Brar schreef:
>>
>>> On Tue, Jun 15, 2010 at 7:25 PM, Kukjin Kim <[email protected]> wrote:
>>>
>>>> Linus Walleij wrote:
>>>>
>>>>> This creates a DMAengine driver for the ARM PL080/PL081 PrimeCells
>>>>> based on the implementation earlier submitted by Peter Pearse.
>>>>> This is working like a charm for memcpy on the PB11MPCore, but
>>>>> slave DMA to devices is still not working.
>>>>>
>>>>> This DMA controller is used in mostly unmodified form in the ARM
>>>>> RealView and Versatile platforms, in the ST-Ericsson Nomadik, and
>>>>> in the ST SPEAr platform.
>>>>>
>>>>> It has been converted to use the header from the Samsung PL080
>>>>> derivate instead of its own defintions, and can potentially support
>>>>> several controllers in the same system.
>>>>>
>>>>> Cc: Peter Pearse <[email protected]>
>>>>> Cc: Ben Dooks <[email protected]>
>>>>> Cc: Kukjin Kim <[email protected]>
>>>>>
>>>> Looks good, but please give me some time to test on the board(SMDK6410).
>>>> If any problem, let you know. Of course no problem, will ack.
>>>>
>>> Samsung doesn't use the DMA API, so this driver is unlikely to work.
>>>
>> It doesn't indeed, but it could be adapted to be a wrapper around the DMA engine API.
>> Or even better, the drivers could be adapted to use that API.
>>
> I don't particularly like the idea of making Samsung's drivers use the DMA API.
> IMHO the S3C dma api is better atm.
>
At the moment it is better, true. But I do think it'd be better to use a
generic API than an arch-specific one.
I'm not familiar enough with both of them to know what functionality is
currently missing but I do think that once the DMA engine API is up to
speed, it should replace the s3c one.
--
Maurus Cuelenaere
[Viresh]
> On 6/14/2010 7:09 PM, Linus Walleij wrote:
> > Hi Viresh, thanks a lot for reviewing this and I'd be *very* happy if
> > you could give it a spin on
> > the SPEAr as well!
>
> I would be happy too linus, will do it in few weeks, right now we are
> running short of time.
Yeah I know that feeling ... anyway I will probably publish a few
more rounds of this before then.
> > In this case we multiplex the memcpy and slave transfers on the few
> > physical channels we have, but I haven't finally decided how to
> handle this:
> > perhaps we should always set on physical channel aside for memcpy
> > so this won't ever fail, and then this special memcpy device entry
> will help.
> >
> > Ideas? Use cases?
>
> Hmmm. I am not sure, but i think we can't hard code a channel for some
> device.
> All channels should be available with both capabilities. If still there
> are
> some conditions (that you might know), where we need to hard code
> channels
> for devices, then this should come from plat data in some way.
Currently I don't hardcode anything, the physical channels
(on the PL081 only two!) will be multiplexed on a first-come
First-served basis. This is a bit problematic since if I start
a DMAengine memcpy test there is a real battle about the channels...
The memcpy test assumes it will always get a channel, see.
I could queue the transfers waiting for a physical channel to
become available but perhaps that's not so good either.
> I have few more doubts that i wanted to ask. Are following supported in
> your
> driver, we need them in SPEAr:
> - Configure burst size of source or destination.
The PrimeCell extension supports this, do you need that in things
that are not PrimeCells? In that case we need to make them generic.
> - Configure DMA Master for src or dest.
Right now I have an algorithm that will (on the PL080, the PL081
has only one master) try to select AHB1 for the memory and AHB2
for the device by checking if one address is fixed. If both or
none addresses are fixed it will simply select AHB1 for source
and AHB2 for destination.
Please elaborate on what algorithm you need for this!
> - Transfer from Peripheral to Peripheral.
Not supported by DMAengine, but would be easy enough to add.
> - Configure Width of src or dest peripheral.
Part of PrimeCell DMA API.
> - Configure Flow controller of transfer.
Currently only done dynamically with DMA as the master for
Mem2mem, mem2per and per2mem. Mastering from the peripherals
is not supported. Do you have advanced features like that?
Anyway it can be passed in from platform data easily.
> - Some callback for fixing Request line multiplexing just before
> initiating transfer.
This is part of this driver. RealView & versatile have exactly
this problem too.
> - Multiple sg elements in slave_sg transfer. I think it is not
> supported.
No, but can be fixed quite easily.
> - Control for autoincrement of addresses, both in case of memory and
> peripherals.
Right now the engine autoincrements the memory pointers if memory
is source/destination and both on mem2mem.
If you actually have peripherals that need increasing pointers it can
Probably be added.
Yours,
Linus Walleij
[Jassi]
> [Kukjin]
> >> It has been converted to use the header from the Samsung PL080
> >> derivate instead of its own defintions, and can potentially support
> >> several controllers in the same system.
> > Looks good, but please give me some time to test on the
> board(SMDK6410).
> > If any problem, let you know. Of course no problem, will ack.
> Samsung doesn't use the DMA API, so this driver is unlikely to work.
Yeah that was never the intention, the Samsung derivate has one extra
Register and other funny stuff.
I was more hoping on breaking out a subset in the same elegant way
with a generic part and a separate DMAengine as was done with the
PL330... But I need to get it fully working first.
Atleast I used the same include file.
But I have functional UART DMA on the RealView PB11MPCore now!
Yours,
Linus Walleij
????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay? ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m???? ????????I?
On 6/16/2010 1:44 AM, Linus WALLEIJ wrote:
> [Viresh]
>
>> On 6/14/2010 7:09 PM, Linus Walleij wrote:
>> I have few more doubts that i wanted to ask. Are following supported in
>> your
>> driver, we need them in SPEAr:
>> - Configure burst size of source or destination.
>
> The PrimeCell extension supports this, do you need that in things
> that are not PrimeCells? In that case we need to make them generic.
>
>> - Configure DMA Master for src or dest.
>
> Right now I have an algorithm that will (on the PL080, the PL081
> has only one master) try to select AHB1 for the memory and AHB2
PL080 have 2 Masters.
> for the device by checking if one address is fixed. If both or
> none addresses are fixed it will simply select AHB1 for source
> and AHB2 for destination.
>
> Please elaborate on what algorithm you need for this!
In the same way, how other peripheral related data is passed to DMA driver,
(like request lines), we can also pass configuration and control information.
This will provide us with all features requested by me, as most of the
control will be from user only. In SPEAr6xx, Memory is accessible from Master1
only but in SPEAr3xx only from Master 2, similar is the pattern with few
peripherals also and so i need control over DMA channel configuration.
>> - Configure Flow controller of transfer.
>
> Currently only done dynamically with DMA as the master for
> Mem2mem, mem2per and per2mem. Mastering from the peripherals
> is not supported. Do you have advanced features like that?
We have JPEG controller, which acts as a flow controller for JPEG to
mem transfer. (Synopsys JPEG Controller).
>
> Anyway it can be passed in from platform data easily.
>
But platform data will be passed one time only and we will not be able to
do it while transferring data at run time.
[Viresh]
> In the same way, how other peripheral related data is passed to DMA driver,
> (like request lines), we can also pass configuration and control information.
Yep.
> In SPEAr6xx, Memory is accessible from Master1
> only but in SPEAr3xx only from Master 2, similar is the pattern with few
> peripherals also and so i need control over DMA channel configuration.
Oh I had no clue that you could set up your masters like that!
Anyway, I'll attempt to hack in some platform config for how
the AHB masters are assigned, but you'll likely have to patch it
to work with SPEAr in the end.
>> Currently only done dynamically with DMA as the master for
>> Mem2mem, mem2per and per2mem. Mastering from the peripherals
>> is not supported. Do you have advanced features like that?
>
> We have JPEG controller, which acts as a flow controller for JPEG to
> mem transfer. (Synopsys JPEG Controller).
OK that's special..
>> Anyway it can be passed in from platform data easily.
>
> But platform data will be passed one time only and we will not be able to
> do it while transferring data at run time.
Usually there is a very fixed use for each virtual DMA channel (which
have a platform config each), i.e. usually there is only one or two
flow controls per virtual channel. So in this case I guess that
Synopsys JPEG has a virtual channel that always is JPEG->mem with
JPEG as master, so it can actually be in fix platform data?
Anyway, we can probably extended either the way we did for PrimeCells
or in some generic way by adding config commands to the DMAengine,
so I see no road blocker.
Yours,
Linus Walleij
Having just looked at this while trying to undo the DMA API abuses
in the PL011 UART driver, I'm getting rather frustrated with this
code.
What's wrong with the PL08x DMA engine driver? Well, in the PL011
UART driver, you do this:
+static void pl011_dma_tx_callback(void *data)
+{
...
+ /* Refill the TX if the buffer is not empty */
+ if (!uart_circ_empty(xmit)) {
+ ret = pl011_dma_tx_refill(uap);
...
+static int pl011_dma_tx_refill(struct uart_amba_port *uap)
+{
...
+ /* Prepare the scatterlist */
+ desc = chan->device->device_prep_slave_sg(chan,
+ &dmatx->scatter,
+ 1,
+ DMA_TO_DEVICE,
+ DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
...
+ /* Some data to go along to the callback */
+ desc->callback = pl011_dma_tx_callback;
+ desc->callback_param = uap;
Note that this calls the channel device_prep_slave_sg() from the
callback. This seems reasonable.
Right, now let's look at this driver (from the latest kernel):
static void pl08x_tasklet(unsigned long data)
{
...
spin_lock(&plchan->lock);
...
dma_async_tx_callback callback =
plchan->at->tx.callback;
void *callback_param =
plchan->at->tx.callback_param;
...
/*
* Callback to signal completion
*/
if (callback)
callback(callback_param);
Note that the callback is called with the channel lock held.
struct dma_async_tx_descriptor *pl08x_prep_slave_sg(
struct dma_chan *chan, struct scatterlist *sgl,
unsigned int sg_len, enum dma_data_direction direction,
unsigned long flags)
{
...
ret = pl08x_prep_channel_resources(plchan, txd);
if (ret)
return NULL;
/*
* NB: the channel lock is held at this point so tx_submit()
* must be called in direct succession.
*/
XXXXXXXX DEADLOCK XXXXXXXX
Has anyone reviewed the locking in the AMBA PL08x DMA driver?
It also seems to do nothing with the DMA_COMPL_* flags - it's unclear
whether it should, but if a user were to specify one of these flags
and the DMA engine driver ignored it, things would get stuffed as far
as the DMA API goes.
These seem to be some really basic errors - and as such I'm far from
happy with even attempting to use this driver to the point that I'm
thinking about starting again with it.
On Tue, Dec 21, 2010 at 06:20:37PM +0000, Russell King - ARM Linux wrote:
> Having just looked at this while trying to undo the DMA API abuses
> in the PL011 UART driver, I'm getting rather frustrated with this
> code.
Here's another couple of problems:
PL011 driver:
+ dmatx->cookie = desc->tx_submit(desc);
+ if (dma_submit_error(dmatx->cookie)) {
+ /* "Complete" DMA (errorpath) */
+ complete(&dmatx->complete);
+ chan->device->device_control(chan, DMA_TERMINATE_ALL, 0);
+ return dmatx->cookie;
+ }
dmatx->cookie is of type dma_cookie_t, which is a s32.
MMCI driver:
+ dma_cookie_t cookie;
...
+ cookie = desc->tx_submit(desc);
+
+ /* Here overloaded DMA controllers may fail */
+ if (dma_submit_error(cookie))
+ goto unmap_exit;
#define dma_submit_error(cookie) ((cookie) < 0 ? 1 : 0)
So, DMA errors are negative values returned from the tx_submit function.
PL08x tx_submit method:
static dma_cookie_t pl08x_tx_submit(struct dma_async_tx_descriptor *tx)
{
struct pl08x_dma_chan *plchan = to_pl08x_chan(tx->chan);
atomic_inc(&plchan->last_issued);
tx->cookie = atomic_read(&plchan->last_issued);
/* This unlock follows the lock in the prep() function */
spin_unlock_irqrestore(&plchan->lock, plchan->lockflags);
return tx->cookie;
}
So, after 2^31 DMA transactions, the DMA engine layer continues happily
along with the queued DMA operation, but the driver gets a negative
cookie returned, and so bails out.
I'm also left wondering about that atomic_t. It's incremented and
read in a locked region, which will in itself prevent two concurrent
increments. The only other places its used is from dma_tx_status()
where it is only ever read, and on the initialization path. That to
me makes no sense.
I'll see if I can get to looking at the PL08x stuff once I've finished
sorting out the PL011 UART driver DMA support into a state where I'm
happy that it's doing things correctly.
On Tue, Dec 21, 2010 at 06:20:37PM +0000, Russell King - ARM Linux wrote:
> Having just looked at this while trying to undo the DMA API abuses
> in the PL011 UART driver, I'm getting rather frustrated with this
> code.
Right, just tried this on the Versatile PB/926, which has a PL080.
The result is DMA errors. This turns out to be the hard-coding of
which AHB bus is used.
You can't hard-code this information into the driver - it's part of
the bus matrix configuration. On Versatile PB/926, the two AHB
buses have different memory maps - see DUI0224 page 3-13:
* DMA0 (which is DMA AHB M1) has access to the APB peripherals
but not the system memory.
* DMA1 (which is DMA AHB M2) has access to the system memory but
none of the APB peripherals.
Throwing in #ifdef's to sort this out resolves the problem on the PB/926.
Looking at the driver code too, I'm having a hard time understanding it,
probably because of the use of "master" and "slave" - I don't think this
has anything to do with which AHB master is used. The comments against
pl08x_choose_master_bus() seem to imply that it controls which AHB master
is used, but it has no apparant effect on that. I don't think this
function does anything like that anymore.
On Tue, Dec 21, 2010 at 06:20:37PM +0000, Russell King - ARM Linux wrote:
> Having just looked at this while trying to undo the DMA API abuses
> in the PL011 UART driver, I'm getting rather frustrated with this
> code.
>
> What's wrong with the PL08x DMA engine driver? Well, in the PL011
> UART driver, you do this:
>
> +static void pl011_dma_tx_callback(void *data)
> +{
> ...
> + /* Refill the TX if the buffer is not empty */
> + if (!uart_circ_empty(xmit)) {
> + ret = pl011_dma_tx_refill(uap);
> ...
> +static int pl011_dma_tx_refill(struct uart_amba_port *uap)
> +{
> ...
> + /* Prepare the scatterlist */
> + desc = chan->device->device_prep_slave_sg(chan,
> + &dmatx->scatter,
> + 1,
> + DMA_TO_DEVICE,
> + DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
> ...
> + /* Some data to go along to the callback */
> + desc->callback = pl011_dma_tx_callback;
> + desc->callback_param = uap;
>
> Note that this calls the channel device_prep_slave_sg() from the
> callback. This seems reasonable.
>
> Right, now let's look at this driver (from the latest kernel):
>
> static void pl08x_tasklet(unsigned long data)
> {
> ...
> spin_lock(&plchan->lock);
> ...
> dma_async_tx_callback callback =
> plchan->at->tx.callback;
> void *callback_param =
> plchan->at->tx.callback_param;
> ...
> /*
> * Callback to signal completion
> */
> if (callback)
> callback(callback_param);
>
> Note that the callback is called with the channel lock held.
>
> struct dma_async_tx_descriptor *pl08x_prep_slave_sg(
> struct dma_chan *chan, struct scatterlist *sgl,
> unsigned int sg_len, enum dma_data_direction direction,
> unsigned long flags)
> {
> ...
> ret = pl08x_prep_channel_resources(plchan, txd);
> if (ret)
> return NULL;
> /*
> * NB: the channel lock is held at this point so tx_submit()
> * must be called in direct succession.
> */
>
> XXXXXXXX DEADLOCK XXXXXXXX
Having found one of my ARM platforms which actually supports DMA (such
a rare thing - neither my Realview EB nor Versatile Express supports it),
I can finally see about run-time checking some of this.
BUG: spinlock lockup on CPU#0, sh/417, c1870a08
Backtrace:
[<c0038880>] (dump_backtrace+0x0/0x10c) from [<c02c20e0>] (dump_stack+0x18/0x1c) r7:c104e000 r6:c1870a08 r5:00000000 r4:00000000
[<c02c20c8>] (dump_stack+0x0/0x1c) from [<c017b520>] (do_raw_spin_lock+0x118/0x154)
[<c017b408>] (do_raw_spin_lock+0x0/0x154) from [<c02c4b98>] (_raw_spin_lock_irqsave+0x54/0x60)
[<c02c4b44>] (_raw_spin_lock_irqsave+0x0/0x60) from [<c01f5828>] (pl08x_prep_channel_resources+0x718/0x8b4)
r7:c1870a08 r6:00000001 r5:c19c3560 r4:ffd04010
[<c01f5110>] (pl08x_prep_channel_resources+0x0/0x8b4) from [<c01f5bb4>] (pl08x_prep_slave_sg+0x120/0x19c)
[<c01f5a94>] (pl08x_prep_slave_sg+0x0/0x19c) from [<c01be7a0>] (pl011_dma_tx_refill+0x164/0x224)
r8:00000020 r7:c03978c0 r6:c1850000 r5:c1847e00 r4:c1847f40
[<c01be63c>] (pl011_dma_tx_refill+0x0/0x224) from [<c01bf1c8>] (pl011_dma_tx_callback+0x7c/0xc4)
[<c01bf14c>] (pl011_dma_tx_callback+0x0/0xc4) from [<c01f4d34>] (pl08x_tasklet+0x60/0x368)
r5:c18709a0 r4:00000000
[<c01f4cd4>] (pl08x_tasklet+0x0/0x368) from [<c004d978>] (tasklet_action+0xa0/0x100)
[<c004d8d8>] (tasklet_action+0x0/0x100) from [<c004e01c>] (__do_softirq+0xa0/0x170)
r8:00000006 r7:00000001 r6:00000018 r5:c104e000 r4:00000100
[<c004df7c>] (__do_softirq+0x0/0x170) from [<c004e140>] (irq_exit+0x54/0x9c)
[<c004e0ec>] (irq_exit+0x0/0x9c) from [<c0029080>] (asm_do_IRQ+0x80/0xa0)
[<c0029000>] (asm_do_IRQ+0x0/0xa0) from [<c00345b8>] (__irq_svc+0x38/0xa0)
As described above in my code analysis, pl08x_tasklet takes the spinlock,
calls pl011_dma_tx_callback and eventually back to pl08x_prep_slave_sg
and pl08x_prep_channel_resources which then try to take the spinlock
again, leading to deadlock.
On Wed, Dec 22, 2010 at 4:29 AM, Russell King - ARM Linux
<[email protected]> wrote:
> On Tue, Dec 21, 2010 at 06:20:37PM +0000, Russell King - ARM Linux wrote:
>> Having just looked at this while trying to undo the DMA API abuses
>> in the PL011 UART driver, I'm getting rather frustrated with this
>> code.
>>
>> What's wrong with the PL08x DMA engine driver? ?Well, in the PL011
>> UART driver, you do this:
>>
>> +static void pl011_dma_tx_callback(void *data)
>> +{
>> ...
>> + ? ? ? /* Refill the TX if the buffer is not empty */
>> + ? ? ? if (!uart_circ_empty(xmit)) {
>> + ? ? ? ? ? ? ? ret = pl011_dma_tx_refill(uap);
>> ...
>> +static int pl011_dma_tx_refill(struct uart_amba_port *uap)
>> +{
>> ...
>> + ? ? ? /* Prepare the scatterlist */
>> + ? ? ? desc = chan->device->device_prep_slave_sg(chan,
>> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? &dmatx->scatter,
>> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1,
>> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? DMA_TO_DEVICE,
>> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
>> ...
>> + ? ? ? /* Some data to go along to the callback */
>> + ? ? ? desc->callback = pl011_dma_tx_callback;
>> + ? ? ? desc->callback_param = uap;
>>
>> Note that this calls the channel device_prep_slave_sg() from the
>> callback. ?This seems reasonable.
>>
>> Right, now let's look at this driver (from the latest kernel):
>>
>> static void pl08x_tasklet(unsigned long data)
>> {
>> ...
>> ? ? ? ? spin_lock(&plchan->lock);
>> ...
>> ? ? ? ? ? ? ? ? dma_async_tx_callback callback =
>> ? ? ? ? ? ? ? ? ? ? ? ? plchan->at->tx.callback;
>> ? ? ? ? ? ? ? ? void *callback_param =
>> ? ? ? ? ? ? ? ? ? ? ? ? plchan->at->tx.callback_param;
>> ...
>> ? ? ? ? ? ? ? ? /*
>> ? ? ? ? ? ? ? ? ?* Callback to signal completion
>> ? ? ? ? ? ? ? ? ?*/
>> ? ? ? ? ? ? ? ? if (callback)
>> ? ? ? ? ? ? ? ? ? ? ? ? callback(callback_param);
>>
>> Note that the callback is called with the channel lock held.
>>
>> struct dma_async_tx_descriptor *pl08x_prep_slave_sg(
>> ? ? ? ? ? ? ? ? struct dma_chan *chan, struct scatterlist *sgl,
>> ? ? ? ? ? ? ? ? unsigned int sg_len, enum dma_data_direction direction,
>> ? ? ? ? ? ? ? ? unsigned long flags)
>> {
>> ...
>> ? ? ? ? ret = pl08x_prep_channel_resources(plchan, txd);
>> ? ? ? ? if (ret)
>> ? ? ? ? ? ? ? ? return NULL;
>> ? ? ? ? /*
>> ? ? ? ? ?* NB: the channel lock is held at this point so tx_submit()
>> ? ? ? ? ?* must be called in direct succession.
>> ? ? ? ? ?*/
>>
>> XXXXXXXX DEADLOCK XXXXXXXX
>
> Having found one of my ARM platforms which actually supports DMA (such
> a rare thing - neither my Realview EB nor Versatile Express supports it),
> I can finally see about run-time checking some of this.
>
> BUG: spinlock lockup on CPU#0, sh/417, c1870a08
> Backtrace:
> [<c0038880>] (dump_backtrace+0x0/0x10c) from [<c02c20e0>] (dump_stack+0x18/0x1c) r7:c104e000 r6:c1870a08 r5:00000000 r4:00000000
> [<c02c20c8>] (dump_stack+0x0/0x1c) from [<c017b520>] (do_raw_spin_lock+0x118/0x154)
> [<c017b408>] (do_raw_spin_lock+0x0/0x154) from [<c02c4b98>] (_raw_spin_lock_irqsave+0x54/0x60)
> [<c02c4b44>] (_raw_spin_lock_irqsave+0x0/0x60) from [<c01f5828>] (pl08x_prep_channel_resources+0x718/0x8b4)
> ?r7:c1870a08 r6:00000001 r5:c19c3560 r4:ffd04010
> [<c01f5110>] (pl08x_prep_channel_resources+0x0/0x8b4) from [<c01f5bb4>] (pl08x_prep_slave_sg+0x120/0x19c)
> [<c01f5a94>] (pl08x_prep_slave_sg+0x0/0x19c) from [<c01be7a0>] (pl011_dma_tx_refill+0x164/0x224)
> ?r8:00000020 r7:c03978c0 r6:c1850000 r5:c1847e00 r4:c1847f40
> [<c01be63c>] (pl011_dma_tx_refill+0x0/0x224) from [<c01bf1c8>] (pl011_dma_tx_callback+0x7c/0xc4)
> [<c01bf14c>] (pl011_dma_tx_callback+0x0/0xc4) from [<c01f4d34>] (pl08x_tasklet+0x60/0x368)
> ?r5:c18709a0 r4:00000000
> [<c01f4cd4>] (pl08x_tasklet+0x0/0x368) from [<c004d978>] (tasklet_action+0xa0/0x100)
> [<c004d8d8>] (tasklet_action+0x0/0x100) from [<c004e01c>] (__do_softirq+0xa0/0x170)
> ?r8:00000006 r7:00000001 r6:00000018 r5:c104e000 r4:00000100
> [<c004df7c>] (__do_softirq+0x0/0x170) from [<c004e140>] (irq_exit+0x54/0x9c)
> [<c004e0ec>] (irq_exit+0x0/0x9c) from [<c0029080>] (asm_do_IRQ+0x80/0xa0)
> [<c0029000>] (asm_do_IRQ+0x0/0xa0) from [<c00345b8>] (__irq_svc+0x38/0xa0)
>
> As described above in my code analysis, pl08x_tasklet takes the spinlock,
> calls pl011_dma_tx_callback and eventually back to pl08x_prep_slave_sg
> and pl08x_prep_channel_resources which then try to take the spinlock
> again, leading to deadlock.
This is listed in the dmaengine documentation [1], but I obviously
missed this before merging. This also would have been caught by
lockdep as required by SubmitChecklist. As far as corrective action
before 2.6.37-final. It looks like this driver needs a full scrub
which seems unreasonable to complete and test over the holidays before
.37 lands. Linus we either need to mark this "depends on BROKEN" or
revert it.
Support for the DMA_COMPL flags are necessary if the DMA_MEMCPY
capability is advertised, yes this driver got this wrong. I'll update
the documentation to make this requirement clear, and audit the other
drivers. With slave-only drivers the only usage model is one where
the client driver owns dma-mapping. In the non-slave (opportunistic
memcpy offload) case the client is unaware of the engine so the driver
owns unmapping. The minimal fix is to disable memcpy offload.
--
Dan
[1]
3.6 Constraints:
1/ Calls to async_<operation> are not permitted in IRQ context. Other
contexts are permitted provided constraint #2 is not violated.
2/ Completion callback routines cannot submit new operations. This
results in recursion in the synchronous case and spin_locks being
acquired twice in the asynchronous case.
On Wed, Dec 22, 2010 at 03:45:39PM -0800, Dan Williams wrote:
> This is listed in the dmaengine documentation [1], but I obviously
> missed this before merging. This also would have been caught by
> lockdep as required by SubmitChecklist. As far as corrective action
> before 2.6.37-final. It looks like this driver needs a full scrub
> which seems unreasonable to complete and test over the holidays before
> .37 lands. Linus we either need to mark this "depends on BROKEN" or
> revert it.
>
> Support for the DMA_COMPL flags are necessary if the DMA_MEMCPY
> capability is advertised, yes this driver got this wrong. I'll update
> the documentation to make this requirement clear, and audit the other
> drivers. With slave-only drivers the only usage model is one where
> the client driver owns dma-mapping. In the non-slave (opportunistic
> memcpy offload) case the client is unaware of the engine so the driver
> owns unmapping. The minimal fix is to disable memcpy offload.
>
> --
> Dan
>
> [1]
> 3.6 Constraints:
> 1/ Calls to async_<operation> are not permitted in IRQ context. Other
> contexts are permitted provided constraint #2 is not violated.
> 2/ Completion callback routines cannot submit new operations. This
> results in recursion in the synchronous case and spin_locks being
> acquired twice in the asynchronous case.
(2) seems to be more than a little annoying - it seems that DMA engine
drivers use a tasklet for running their DMA cleanup, which calls drivers
callbacks, and we're going to have to have a whole pile of taskets
in drivers just to be triggered from the completion callback. I can
see this adding an additional layer of complexity and a nice fine set
of shiney new races to deal with.
On Wed, Dec 22, 2010 at 03:45:39PM -0800, Dan Williams wrote:
> 3.6 Constraints:
> 1/ Calls to async_<operation> are not permitted in IRQ context. Other
> contexts are permitted provided constraint #2 is not violated.
BTW, this is misleading. Have the functions been renamed dma_async_xxx(),
eg dma_async_memcpy_buf_to_buf etc, or are you referring just to:
async_dmaengine_get
async_dmaengine_put
async_dma_find_channel
async_dma_find_channel
async_tx_ack
async_tx_clear_ack
async_tx_test_ack
Beware of just renaming it to dma_async_<operation> as there's other
functions in that namespace which may not be appropriate.
Eg, is it really illegal to issue call dma_async_issue_pending() from
IRQ context? That'd make it exceedingly difficult to use the DMA
engine with the slave API in a lot of device drivers.
On Wed, Dec 22, 2010 at 3:54 PM, Russell King - ARM Linux
<[email protected]> wrote:
> On Wed, Dec 22, 2010 at 03:45:39PM -0800, Dan Williams wrote:
>> This is listed in the dmaengine documentation [1], but I obviously
>> missed this before merging. ?This also would have been caught by
>> lockdep as required by SubmitChecklist. ?As far as corrective action
>> before 2.6.37-final. ?It looks like this driver needs a full scrub
>> which seems unreasonable to complete and test over the holidays before
>> .37 lands. ?Linus we either need to mark this "depends on BROKEN" or
>> revert it.
>>
>> Support for the DMA_COMPL flags are necessary if the DMA_MEMCPY
>> capability is advertised, yes this driver got this wrong. ?I'll update
>> the documentation to make this requirement clear, and audit the other
>> drivers. ?With slave-only drivers the only usage model is one where
>> the client driver owns dma-mapping. ?In the non-slave (opportunistic
>> memcpy offload) case the client is unaware of the engine so the driver
>> owns unmapping. ?The minimal fix is to disable memcpy offload.
>>
>> --
>> Dan
>>
>> [1]
>> 3.6 Constraints:
>> 1/ Calls to async_<operation> are not permitted in IRQ context. ?Other
>> ? ?contexts are permitted provided constraint #2 is not violated.
>> 2/ Completion callback routines cannot submit new operations. ?This
>> ? ?results in recursion in the synchronous case and spin_locks being
>> ? ?acquired twice in the asynchronous case.
>
> (2) seems to be more than a little annoying - it seems that DMA engine
> drivers use a tasklet for running their DMA cleanup, which calls drivers
> callbacks, and we're going to have to have a whole pile of taskets
> in drivers just to be triggered from the completion callback. ?I can
> see this adding an additional layer of complexity and a nice fine set
> of shiney new races to deal with.
I should clarify, this is the async_memcpy() api requirement which is
not used outside of md/raid5. DMA drivers can and do allow new
submissions from callbacks, and the ones that do so properly move the
callback outside of the driver lock. The doc needs updating to
reflect present reality, but it at least should have prompted the same
reaction you had when reading it and triggered a question about how to
support that usage model.
--
Dan
On Wed, Dec 22, 2010 at 4:10 PM, Russell King - ARM Linux
<[email protected]> wrote:
> On Wed, Dec 22, 2010 at 03:45:39PM -0800, Dan Williams wrote:
>> 3.6 Constraints:
>> 1/ Calls to async_<operation> are not permitted in IRQ context. ?Other
>> ? ?contexts are permitted provided constraint #2 is not violated.
>
> BTW, this is misleading. ?Have the functions been renamed dma_async_xxx(),
> eg dma_async_memcpy_buf_to_buf etc, or are you referring just to:
>
> ? ? ? ?async_dmaengine_get
> ? ? ? ?async_dmaengine_put
> ? ? ? ?async_dma_find_channel
> ? ? ? ?async_dma_find_channel
> ? ? ? ?async_tx_ack
> ? ? ? ?async_tx_clear_ack
> ? ? ? ?async_tx_test_ack
>
> Beware of just renaming it to dma_async_<operation> as there's other
> functions in that namespace which may not be appropriate.
>
> Eg, is it really illegal to issue call dma_async_issue_pending() from
> IRQ context? ?That'd make it exceedingly difficult to use the DMA
> engine with the slave API in a lot of device drivers.
This is generic offload (async_{memcpy|xor|etc...}) versus the slave
usage confusion . In the async_<operation> case the client (md/raid5)
has no idea if a dmaengine is offloading the operation behind the
scenes and should not make any assumptions about submission context
beyond what is allowed in the document. In the slave case the client
driver knows that it is talking to a specific dma driver. The
contract between the client driver and dma driver is undocumented.
The slave usage model really is a "I want to use dmaengine to find my
dma device driver / manage channels, and I want a common prep / submit
mechanism, but otherwise stay out of my way". Drivers that do not
want to meet the constraints expected by the opportunistic offload
clients should do what ste_dma40 does and add "depends on !(NET_DMA ||
ASYNC_TX_DMA)"
--
Dan
On Wed, Dec 22, 2010 at 5:11 PM, Dan Williams <[email protected]> wrote:
> Drivers that do not
> want to meet the constraints expected by the opportunistic offload
> clients should do what ste_dma40 does and add "depends on !(NET_DMA ||
> ASYNC_TX_DMA)"
Sorry I was looking at a local change it does not do this today, but I
recommended it to workaround the broken >64k transfer support that was
reported recently.
--
Dan
2010/12/23 Dan Williams <[email protected]>:
> On Wed, Dec 22, 2010 at 4:29 AM, Russell King - ARM Linux
> <[email protected]> wrote:
>> As described above in my code analysis, pl08x_tasklet takes the spinlock,
>> calls pl011_dma_tx_callback and eventually back to pl08x_prep_slave_sg
>> and pl08x_prep_channel_resources which then try to take the spinlock
>> again, leading to deadlock.
>
> This is listed in the dmaengine documentation [1], but I obviously
> missed this before merging. ?This also would have been caught by
> lockdep as required by SubmitChecklist.
Yeah, my bad. I'll get better at this... :-(
(I blame it partially on inaccessible hardware, sob sob. I do like to
run lockdep.)
>?It looks like this driver needs a full scrub
> which seems unreasonable to complete and test over the holidays before
> .37 lands. ?Linus we either need to mark this "depends on BROKEN" or
> revert it.
Isn't it really as simple as to release the spinlock during callbacks?
That lock is only intended to protect the plchan variables, not to block
anyone from queueing new stuff during the callback (as happens now).
It can release that lock, make a callback where a new descriptor
gets queued, and then take it again and start looking at the queue,
at which point it discovers the new desc and process it.
So something like this:
From: Linus Walleij <[email protected]>
Date: Thu, 23 Dec 2010 09:06:14 +0100
Subject: [PATCH] dma: release pl08x channel lock during callback
The spinlock is not really safeguarding any resources during the
callback, so let's release it before that and take it back
afterwards so as to avoid deadlocks.
Signed-off-by: Linus Walleij <[email protected]>
---
drivers/dma/amba-pl08x.c | 7 +++++--
1 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/drivers/dma/amba-pl08x.c b/drivers/dma/amba-pl08x.c
index b605cc9..7879a22 100644
--- a/drivers/dma/amba-pl08x.c
+++ b/drivers/dma/amba-pl08x.c
@@ -1651,8 +1651,11 @@ static void pl08x_tasklet(unsigned long data)
/*
* Callback to signal completion
*/
- if (callback)
- callback(callback_param);
+ if (callback) {
+ spin_unlock(&plchan->lock);
+ callback(callback_param);
+ spin_lock(&plchan->lock);
+ }
/*
* Device callbacks should NOT clear
--
1.7.2.3
Yours,
Linus Walleij
On Thu, Dec 23, 2010 at 5:17 PM, Linus Walleij
<[email protected]> wrote:
> 2010/12/23 Dan Williams <[email protected]>:
>> On Wed, Dec 22, 2010 at 4:29 AM, Russell King - ARM Linux
>> <[email protected]> wrote:
>>> As described above in my code analysis, pl08x_tasklet takes the spinlock,
>>> calls pl011_dma_tx_callback and eventually back to pl08x_prep_slave_sg
>>> and pl08x_prep_channel_resources which then try to take the spinlock
>>> again, leading to deadlock.
>>
>> This is listed in the dmaengine documentation [1], but I obviously
>> missed this before merging. This also would have been caught by
>> lockdep as required by SubmitChecklist.
>
> Yeah, my bad. I'll get better at this... :-(
> (I blame it partially on inaccessible hardware, sob sob. I do like to
> run lockdep.)
>
>> It looks like this driver needs a full scrub
>> which seems unreasonable to complete and test over the holidays before
>> .37 lands. Linus we either need to mark this "depends on BROKEN" or
>> revert it.
>
> Isn't it really as simple as to release the spinlock during callbacks?
> That lock is only intended to protect the plchan variables, not to block
> anyone from queueing new stuff during the callback (as happens now).
>
> It can release that lock, make a callback where a new descriptor
> gets queued, and then take it again and start looking at the queue,
> at which point it discovers the new desc and process it.
>
> So something like this:
>
>
> From: Linus Walleij <[email protected]>
> Date: Thu, 23 Dec 2010 09:06:14 +0100
> Subject: [PATCH] dma: release pl08x channel lock during callback
>
> The spinlock is not really safeguarding any resources during the
> callback, so let's release it before that and take it back
> afterwards so as to avoid deadlocks.
>
> Signed-off-by: Linus Walleij <[email protected]>
> ---
> drivers/dma/amba-pl08x.c | 7 +++++--
> 1 files changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/dma/amba-pl08x.c b/drivers/dma/amba-pl08x.c
> index b605cc9..7879a22 100644
> --- a/drivers/dma/amba-pl08x.c
> +++ b/drivers/dma/amba-pl08x.c
> @@ -1651,8 +1651,11 @@ static void pl08x_tasklet(unsigned long data)
> /*
> * Callback to signal completion
> */
> - if (callback)
> - callback(callback_param);
> + if (callback) {
> + spin_unlock(&plchan->lock);
> + callback(callback_param);
> + spin_lock(&plchan->lock);
> + }
How about adding completed requests to a list and go on to do important
channel management stuff, and do callbacks at the end after dropping the lock.
As in pl330_tasklet of drivers/dma/pl330.c
On Wed, Dec 22, 2010 at 05:11:24PM -0800, Dan Williams wrote:
> On Wed, Dec 22, 2010 at 4:10 PM, Russell King - ARM Linux
> <[email protected]> wrote:
> > On Wed, Dec 22, 2010 at 03:45:39PM -0800, Dan Williams wrote:
> >> 3.6 Constraints:
> >> 1/ Calls to async_<operation> are not permitted in IRQ context. ?Other
> >> ? ?contexts are permitted provided constraint #2 is not violated.
> >
> > BTW, this is misleading. ?Have the functions been renamed dma_async_xxx(),
> > eg dma_async_memcpy_buf_to_buf etc, or are you referring just to:
> >
> > ? ? ? ?async_dmaengine_get
> > ? ? ? ?async_dmaengine_put
> > ? ? ? ?async_dma_find_channel
> > ? ? ? ?async_dma_find_channel
> > ? ? ? ?async_tx_ack
> > ? ? ? ?async_tx_clear_ack
> > ? ? ? ?async_tx_test_ack
> >
> > Beware of just renaming it to dma_async_<operation> as there's other
> > functions in that namespace which may not be appropriate.
> >
> > Eg, is it really illegal to issue call dma_async_issue_pending() from
> > IRQ context? ?That'd make it exceedingly difficult to use the DMA
> > engine with the slave API in a lot of device drivers.
>
> This is generic offload (async_{memcpy|xor|etc...}) versus the slave
> usage confusion.
My point is that async_<operation> refers to something that doesn't
exist. What we do have:
dma_cookie_t dma_async_memcpy_buf_to_buf(struct dma_chan *chan,
void *dest, void *src, size_t len);
dma_cookie_t dma_async_memcpy_buf_to_pg(struct dma_chan *chan,
struct page *page, unsigned int offset, void *kdata, size_t len);
dma_cookie_t dma_async_memcpy_pg_to_pg(struct dma_chan *chan,
struct page *dest_pg, unsigned int dest_off, struct page *src_pg,
unsigned int src_off, size_t len);
Nothing in dmaengine.h matches 'async_xor', so that suggests it's
incomplete. There's another problem with referring to them as
dma_async_<operation> because we also have:
static inline void dma_async_issue_pending(struct dma_chan *chan)
#define dma_async_memcpy_issue_pending(chan) dma_async_issue_pending(chan)
static inline enum dma_status dma_async_is_tx_complete(struct dma_chan *chan,
dma_cookie_t cookie, dma_cookie_t *last, dma_cookie_t *used)
#define dma_async_memcpy_complete(chan, cookie, last, used)\
static inline enum dma_status dma_async_is_complete(dma_cookie_t cookie,
dma_cookie_t last_complete, dma_cookie_t last_used)
int dma_async_device_register(struct dma_device *device);
void dma_async_device_unregister(struct dma_device *device);
So just referring to dma_async_<operation> will also be confusing.
There's also a bunch of other async_* named functions which makes it
look like these are what's being referred to (see the list above).
So, the documentation is confusing.
Maybe it would be better to refer to struct dma_device's device_prep_dma_*
methods and anything which calls them, with the exception of
device_prep_dma_cyclic? (Shouldn't that be device_prep_slave_cyclic?)
On Thu, Dec 23, 2010 at 09:17:07AM +0100, Linus Walleij wrote:
> 2010/12/23 Dan Williams <[email protected]>:
> >?It looks like this driver needs a full scrub
> > which seems unreasonable to complete and test over the holidays before
> > .37 lands. ?Linus we either need to mark this "depends on BROKEN" or
> > revert it.
>
> Isn't it really as simple as to release the spinlock during callbacks?
> That lock is only intended to protect the plchan variables, not to block
> anyone from queueing new stuff during the callback (as happens now).
>
> It can release that lock, make a callback where a new descriptor
> gets queued, and then take it again and start looking at the queue,
> at which point it discovers the new desc and process it.
Is it actually safe to do this? The answer seems to be no - if we happen
to terminate all transfers (as your PL011 uart code does) when we fail to
setup a new DMA transaction, then bad stuff happens due to this:
/*
* Device callbacks should NOT clear
* the current transaction on the channel
* Linus: sometimes they should?
*/
if (!plchan->at)
BUG();
We really need a saner approach here - maybe the list approach described
by Jassie.
> diff --git a/drivers/dma/amba-pl08x.c b/drivers/dma/amba-pl08x.c
> index b605cc9..7879a22 100644
> --- a/drivers/dma/amba-pl08x.c
> +++ b/drivers/dma/amba-pl08x.c
> @@ -1651,8 +1651,11 @@ static void pl08x_tasklet(unsigned long data)
> /*
> * Callback to signal completion
> */
> - if (callback)
> - callback(callback_param);
> + if (callback) {
> + spin_unlock(&plchan->lock);
> + callback(callback_param);
> + spin_lock(&plchan->lock);
Plus, of course, that tasklets run with IRQs enabled. This means we're
taking this spinlock in an interruptible context. If we have some other
path which also takes this lock from an IRQ context, then we're asking
for deadlock. See my previous mails on this subject.
I'm currently splitting my dirty patch, and attacking this driver to
clean up some of it into a more reasonable shape, so this is one area
which I'm going to be sorting out.
Patches later.
2010/12/23 Russell King - ARM Linux <[email protected]>:
> I'm currently splitting my dirty patch, and attacking this driver to
> clean up some of it into a more reasonable shape, so this is one area
> which I'm going to be sorting out.
Thanks Russell, much appreciated. I'll be quick to review and ACK
any patches hopefully.
Yours,
Linus Walleij
On Wed, Dec 22, 2010 at 05:31:25PM -0800, Dan Williams wrote:
> On Wed, Dec 22, 2010 at 5:11 PM, Dan Williams <[email protected]> wrote:
> > Drivers that do not
> > want to meet the constraints expected by the opportunistic offload
> > clients should do what ste_dma40 does and add "depends on !(NET_DMA ||
> > ASYNC_TX_DMA)"
>
> Sorry I was looking at a local change it does not do this today, but I
> recommended it to workaround the broken >64k transfer support that was
> reported recently.
Dan,
Is there any documentation on the DMA engine APIs than what's in
crypto/async-tx-api.txt?
Reason for asking is that there's no way at the moment to tell what the
expectations are from a lot of the DMA engine support code - and that is
_very_ bad news if you want DMA engine drivers to behave the same.
I can already see that drivers on both sides of the DMA engine API have
different expectations between their respective implementations, and this
is just adding to confusion.
For instance, the sequence in a driver:
desc = dmadev->device_prep_slave_sg(chan, ...);
if (!desc)
/* what DMA engine cleanup is expected here? */
cookie = dmaengine_submit(desc);
if (dma_submit_error(cookie))
/* what DMA engine cleanup of desc is expected here? */
Note: I don't trust what's written in 3.3 of async-tx-api.txt, because
that seems to be talking about the the async_* APIs rather than the
DMA engine API. (see below.)
1. Is it valid to call dmaengine_terminate_all(chan) in those paths?
2. What is the expectations wrt the callback of a previously submitted
job at the point that dmaengine_terminate_all() returns?
3. What if the callback is running on a different CPU, waiting for a
spinlock you're holding at the time you call dmaengine_terminate_all()
within that very same spinlock?
4. What if dmaengine_terminate_all() is running, but parallel with it
the tasklet runs on a different CPU, and queues another DMA job?
These can all be solved by requiring that the termination implementations
call tasklet_disable(), then clean up the DMA state, followed by
tasklet_enable(). tasklet_disable() will prevent the tasklet being
scheduled, and wait for the tasklet to finish running before proceeding.
This means that (2) becomes "the tasklet will not be running", (3)
becomes illegal (due to deadlock), and (4) becomes predictable as we
know that after tasklet_disable() we have consistent DMA engine state
and we can terminate everything that's been queued.
That still leaves the issue of (1), and also what cleanup is expected.
I'm not entirely clear about the usage of DMA_CTRL_ACK:
* @DMA_CTRL_ACK - if clear, the descriptor cannot be reused until the client
* acknowledges receipt, i.e. has has a chance to establish any dependency
* chains
Some DMA engine using drivers set DMA_CTRL_ACK, others don't.
Should drivers using prep_slave_sg() be requesting their descriptors
with DMA_CTRL_ACK in the flags argument? Doesn't that mean that the
DMA engine driver is free to re-use this descriptor beneath the driver?
Almost no one checks the result of dmaengine_submit() (or its open-coded
equivalent). Are all such drivers potentially leaking descriptors? If
not, how are the failed descriptors re-used?
Also, I think that the DMA engine core code needs to provide some
essential helper functions to prevent buggy bodgerations such as
what's happening in amba-pl08x.c, such as:
dma_cookie_t dmaengine_alloc_cookie(struct dma_async_tx_descriptor *tx)
{
struct dma_chan *chan = tx->chan;
chan->cookie += 1;
if (chan->cookie < 0)
chan->cookie = 1;
return (tx->cookie = chan->cookie);
}
What should be the initial value of tx->cookie after a successful
prep_slave_sg() call?
Also a helper function for dmaengine drivers to call when a descriptor
is complete to handle all the tx descriptor cleanup on completion, so
that all dmaengine drivers don't have to re-implement the cleanup in
their own ways, each with differing behaviour. (Can the TX desciptor
structure be expanded to hold all the information needed so that core
code can implement the DMA unmapping for the asynctx stuff there?)
I think that's enough to think about for the time being - I'm sure
there's lots more...
As it is, even if I thought trying to fix the PL08x driver was worth the
effort (in spite of the hardware issues), I don't think there's enough
documentation on the DMA engine API to be able to do so at the present
time.
So, given all these questions (some of which can lead to deadlocks), and
we're now at -rc8, I see no way that I can sanely (or safely) queue up
the PL011 UART and PL180 MMCI DMA engine code for this coming merge
window. (Sorry Linus.)