2009-01-07 20:10:33

by Ira W. Snyder

Subject: [PATCH RFC v5] net: add PCINet driver

This adds support to Linux for a virtual ethernet interface which uses the
PCI bus as its transport mechanism. It creates a simple, familiar, and fast
method of communication for two devices connected by a PCI interface.

I have implemented client support for the Freescale MPC8349EMDS board,
which is capable of running in PCI Agent mode (It acts like a PCI card, but
is a complete PowerPC computer, running Linux). It is almost certainly
trivially ported to any MPC83xx system.

It was developed to work in a CompactPCI crate of computers, one of which
is a relatively standard x86 system (acting as the host) and many PowerPC
systems (acting as clients).

I have tested with two boards in my system. The host is a 1066MHz
Pentium3-M. I am able to achieve the following bandwidth, as measured with
netperf's TCP_STREAM test. Some improvement would be nice. I'm out of
ideas and need suggestions at this point.

Host -> Board1 (Board2 idle): 18 MB/sec
Host -> Board1, Board2: 18 MB/sec
Board1 -> Host (Board2 idle): 32 MB/sec
Board1, Board2 -> Host: 44 MB/sec

RFC v4 -> RFC v5:
* convert to net_device_ops infrastructure
* fix fallout from NAPI API changes
* use seperate DMA channels for RX and TX
* use skb_dma_map()/skb_dma_unmap() infrastructure
* unify the RX and TX DMA routines
* fix uart allocation with multiple boards in a single system
* mask interrupts when not in use
* remove status bits, relying on the state of interrupts instead
* fix circular locking violation reported by the locking validator

RFC v3 -> RFC v4:
* dropped NETIF_F_NO_CSUM from struct net_device->features, it was
causing nfsroot to fail due to UDP checksum errors

RFC v2 -> RFC v3:
* use inline functions for accessing struct circ_buf_desc
* use pointer dereferencing on PowerPC local memory instead of ioread32()
* move IMMR and buffer descriptor accessors inside drivers
* update for dma_mapping_error() API changes
* use minimal locking primitives (i.e. spin_lock() instead of _irqsave())
* always disable checksumming, PCI is reliable
* replace typedef cbd_t with struct circ_buf_desc
* use get_immrbase() to get IMMR register offsets

RFC v1 -> RFC v2:
* remove vim modelines
* use net_device->name in request_irq(), for irqbalance
* remove unneccesary wqt_get_stats(), use default get_stats() instead
* use dev_printk() and friends
* add message unit to MPC8349EMDS dts file

Signed-off-by: Ira W. Snyder <[email protected]>
---
This is the fifth RFC posting of this driver. I haven't gotten much
feedback recently. I'd really like to see some version of this go
upstream, so I appreciate your review.

Thanks to all of those who have posted comments about the driver.

Remaining Issues:
1) Naming
The name wqt originally stood for "workqueue test". That test driver
eventually evolved into this one. I'm open to suggestions for names.
2) IMMR Usage
In the Freescale client driver, I ioremap() the whole set of board
control registers, even though I don't make use of most of them. This
was done so that the register offsets could be shared between the drivers
in pcinet_hw.h
3) Hardcoded DMA Window Address
In the Freescale client driver, the DMA window address (0x80000000) is
hardcoded into the DMA transfer code. Suggestions on how to get this
value at runtime are welcome (if it matters).
4) Speed
This driver is relatively fast, but not quite fast enough for my
application. I'm out of ideas, and I need some help speeding this up.

Rationale behind some decisions:
1) Buffer Descriptors in client memory
I chose to put the buffer descriptors in client memory rather than host
memory. It seemed more logical to me at the time. In my usage of this
driver, I'll have 19 boards + 1 host per cPCI chassis. I thought this
would cut down on the number of PCI accesses needed. I'm willing to
make changes.
2) Usage of client DMA controller for all data transfers
This was done purely for speed. I tried using the CPU to transfer all
data, and it is very slow: about 3MB/sec. Using the DMA controller
brings the transfer rate up to about 30MB/sec.
3) Static 1GB DMA window
Maintaining changing DMA windows while DMA are in flight seemed like
it would overcomplicate the design.
4) Serial Driver
Unfortunately, there are two drivers here. I needed a method to
with the U-Boot bootloader on these boards without plugging in a physical
cable. With 19 clients + 1 host per chassis, the cable clutter is worth
avoiding.

I'll post both U-Boot drivers to their mailing list once this driver is
finalized.

Thanks,
Ira

arch/powerpc/boot/dts/mpc834x_mds.dts | 7 +
drivers/net/Kconfig | 29 +
drivers/net/Makefile | 3 +
drivers/net/pcinet.h | 62 ++
drivers/net/pcinet_fsl.c | 1335 +++++++++++++++++++++++++++++++++
drivers/net/pcinet_host.c | 1318 ++++++++++++++++++++++++++++++++
drivers/net/pcinet_hw.h | 77 ++
7 files changed, 2831 insertions(+), 0 deletions(-)
create mode 100644 drivers/net/pcinet.h
create mode 100644 drivers/net/pcinet_fsl.c
create mode 100644 drivers/net/pcinet_host.c
create mode 100644 drivers/net/pcinet_hw.h

diff --git a/arch/powerpc/boot/dts/mpc834x_mds.dts b/arch/powerpc/boot/dts/mpc834x_mds.dts
index d9adba0..5c7617d 100644
--- a/arch/powerpc/boot/dts/mpc834x_mds.dts
+++ b/arch/powerpc/boot/dts/mpc834x_mds.dts
@@ -104,6 +104,13 @@
mode = "cpu";
};

+ message-unit@8030 {
+ compatible = "fsl,mpc8349-mu";
+ reg = <0x8030 0xd0>;
+ interrupts = <69 0x8>;
+ interrupt-parent = <&ipic>;
+ };
+
dma@82a8 {
#address-cells = <1>;
#size-cells = <1>;
diff --git a/drivers/net/Kconfig b/drivers/net/Kconfig
index 9a18270..3a8ad4d 100644
--- a/drivers/net/Kconfig
+++ b/drivers/net/Kconfig
@@ -2298,6 +2298,35 @@ config UGETH_TX_ON_DEMAND
bool "Transmit on Demand support"
depends on UCC_GETH

+config PCINET_FSL
+ tristate "PCINet Virtual Ethernet over PCI support (Freescale)"
+ depends on MPC834x_MDS
+ select DMA_ENGINE
+ select FSL_DMA
+ help
+ When running as a PCI Agent, this driver will create a virtual
+ ethernet link running over the PCI bus, allowing simplified
+ communication with the host system. The host system will need
+ to use the corresponding driver.
+
+ If in doubt, say N.
+
+config PCINET_HOST
+ tristate "PCINet Virtual Ethernet over PCI support (Host)"
+ depends on PCI
+ help
+ This driver will let you communicate with a PCINet client device
+ using a virtual ethernet link running over the PCI bus. This
+ allows simplified communication with the client system.
+
+ This is inteded for use in a system that has a crate full of
+ computers running Linux, all connected by a PCI backplane.
+
+ The currently, the only board implementing client support is the
+ Freescale MPC8349EMDS. Enable support with CONFIG_PCINET_FSL=y.
+
+ If in doubt, say N.
+
config MV643XX_ETH
tristate "Marvell Discovery (643XX) and Orion ethernet support"
depends on MV64360 || MV64X60 || (PPC_MULTIPLATFORM && PPC32) || PLAT_ORION
diff --git a/drivers/net/Makefile b/drivers/net/Makefile
index e5c34b4..664849a 100644
--- a/drivers/net/Makefile
+++ b/drivers/net/Makefile
@@ -30,6 +30,9 @@ gianfar_driver-objs := gianfar.o \
obj-$(CONFIG_UCC_GETH) += ucc_geth_driver.o
ucc_geth_driver-objs := ucc_geth.o ucc_geth_mii.o ucc_geth_ethtool.o

+obj-$(CONFIG_PCINET_FSL) += pcinet_fsl.o
+obj-$(CONFIG_PCINET_HOST) += pcinet_host.o
+
#
# link order important here
#
diff --git a/drivers/net/pcinet.h b/drivers/net/pcinet.h
new file mode 100644
index 0000000..ea1f806
--- /dev/null
+++ b/drivers/net/pcinet.h
@@ -0,0 +1,62 @@
+/*
+ * Shared Definitions for the PCINet / PCISerial drivers
+ *
+ * Copyright (c) 2008 Ira W. Snyder <[email protected]>
+ *
+ * Heavily inspired by the drivers/net/fs_enet driver
+ *
+ * This file is licensed under the terms of the GNU General Public License
+ * version 2. This program is licensed "as is" without any warranty of any
+ * kind, whether express or implied.
+ */
+
+#ifndef PCINET_H
+#define PCINET_H
+
+#include <linux/kernel.h>
+#include <linux/if_ether.h>
+
+/* Ring and Frame size -- these must match between the drivers */
+#define PH_RING_SIZE (64)
+#define PH_MAX_FRSIZE (64 * 1024)
+#define PH_MAX_MTU (PH_MAX_FRSIZE - ETH_HLEN)
+
+struct circ_buf_desc {
+ __le32 sc;
+ __le32 len;
+ __le32 addr;
+ __le32 padding; /* unused padding */
+};
+
+/* Buffer Descriptor Registers */
+#define PCINET_TXBD_BASE 0x400
+#define PCINET_RXBD_BASE 0x800
+
+/* Buffer Descriptor Status */
+#define BD_MEM_READY 0x1
+#define BD_MEM_DIRTY 0x2
+#define BD_MEM_FREE 0x3
+
+/* Status Register Bits */
+#define PCINET_UART_RX_ENABLED (1<<0)
+#define PCINET_NET_STATUS_RUNNING (1<<1)
+#define PCINET_NET_RXINT_OFF (1<<2)
+#define PCINET_NET_REGISTERS_VALID (1<<3)
+
+/* Driver State Bits */
+#define NET_STATE_STOPPED 0
+#define NET_STATE_RUNNING 1
+
+/* Doorbell Registers */
+#define UART_RX_READY_DBELL (1<<0)
+#define UART_TX_EMPTY_DBELL (1<<1)
+#define NET_RX_PACKET_DBELL (1<<2)
+#define NET_TX_COMPLETE_DBELL (1<<3)
+#define NET_START_REQ_DBELL (1<<4)
+#define NET_START_ACK_DBELL (1<<5)
+#define NET_STOP_REQ_DBELL (1<<6)
+#define NET_STOP_ACK_DBELL (1<<7)
+#define NET_ENABLE_RX_PACKET_DBELL (1<<8)
+#define NET_DISABLE_RX_PACKET_DBELL (1<<9)
+
+#endif /* PCINET_H */
diff --git a/drivers/net/pcinet_fsl.c b/drivers/net/pcinet_fsl.c
new file mode 100644
index 0000000..3c05e8d
--- /dev/null
+++ b/drivers/net/pcinet_fsl.c
@@ -0,0 +1,1335 @@
+/*
+ * PCINet and PCISerial Driver for Freescale MPC8349EMDS
+ *
+ * Copyright (c) 2008 Ira W. Snyder <[email protected]>
+ *
+ * Heavily inspired by the drivers/net/fs_enet driver
+ *
+ * This file is licensed under the terms of the GNU General Public License
+ * version 2. This program is licensed "as is" without any warranty of any
+ * kind, whether express or implied.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/of_platform.h>
+#include <linux/sched.h>
+#include <linux/wait.h>
+#include <linux/interrupt.h>
+#include <linux/irqreturn.h>
+#include <linux/delay.h>
+#include <linux/io.h>
+#include <linux/serial.h>
+#include <linux/serial_core.h>
+#include <linux/etherdevice.h>
+#include <linux/mutex.h>
+#include <linux/dmaengine.h>
+
+/* MPC8349EMDS specific get_immrbase() */
+#include <sysdev/fsl_soc.h>
+
+#include "pcinet.h"
+#include "pcinet_hw.h"
+
+/* IMMR Accessor Helpers */
+#define IMMR_R32(_off) ioread32(priv->immr+(_off))
+#define IMMR_W32(_off, _val) iowrite32((_val), priv->immr+(_off))
+#define IMMR_R32BE(_off) ioread32be(priv->immr+(_off))
+#define IMMR_W32BE(_off, _val) iowrite32be((_val), priv->immr+(_off))
+
+static const char driver_name[] = "wqt";
+
+static void wqtuart_stop_tx(struct uart_port *port);
+
+struct wqt_dev;
+typedef void (*wqt_irqhandler_t)(struct wqt_dev *);
+
+struct wqt_irqhandlers {
+ wqt_irqhandler_t net_start_req_handler;
+ wqt_irqhandler_t net_start_ack_handler;
+ wqt_irqhandler_t net_stop_req_handler;
+ wqt_irqhandler_t net_stop_ack_handler;
+ wqt_irqhandler_t net_rx_packet_handler;
+ wqt_irqhandler_t net_tx_complete_handler;
+ wqt_irqhandler_t uart_rx_ready_handler;
+ wqt_irqhandler_t uart_tx_empty_handler;
+};
+
+struct wqt_dev {
+ /*--------------------------------------------------------------------*/
+ /* OpenFirmware Infrastructure */
+ /*--------------------------------------------------------------------*/
+ int irq;
+ struct device *dev;
+ void __iomem *immr;
+
+ struct mutex irq_mutex;
+ int interrupt_count;
+
+ spinlock_t irq_lock;
+ struct wqt_irqhandlers handlers;
+
+ /*--------------------------------------------------------------------*/
+ /* UART Device Infrastructure */
+ /*--------------------------------------------------------------------*/
+ struct uart_port port;
+ bool uart_open;
+
+ struct workqueue_struct *wq;
+ struct work_struct uart_tx_work;
+ wait_queue_head_t uart_tx_wait; /* sleep for uart_tx_ready */
+ bool uart_tx_ready; /* transmitter state */
+
+ /*--------------------------------------------------------------------*/
+ /* Ethernet Device Infrastructure */
+ /*--------------------------------------------------------------------*/
+ struct net_device *ndev;
+ void *netregs;
+ dma_addr_t netregs_addr;
+
+ /* Circular Buffer Descriptor base */
+ struct circ_buf_desc *rx_base;
+ struct circ_buf_desc *tx_base;
+
+ /* Current SKB index */
+ struct circ_buf_desc *cur_rx;
+ struct circ_buf_desc *cur_tx;
+ struct circ_buf_desc *dirty_tx;
+ int tx_free;
+
+ struct tasklet_struct tx_complete_tasklet;
+ spinlock_t net_lock;
+
+ struct mutex net_mutex;
+ int net_state;
+ struct work_struct net_start_work;
+ struct work_struct net_stop_work;
+ struct completion net_start_completion;
+ struct completion net_stop_completion;
+ struct napi_struct napi;
+
+ bool net_send_rx_packet_dbell;
+
+ /*--------------------------------------------------------------------*/
+ /* DMA Controller */
+ /*--------------------------------------------------------------------*/
+ struct dma_client client;
+ struct dma_chan *rx_chan;
+ struct dma_chan *tx_chan;
+};
+
+/*----------------------------------------------------------------------------*/
+/* Buffer Descriptor Accessor Helpers */
+/*----------------------------------------------------------------------------*/
+
+static inline void cbd_write(__le32 *addr, u32 val)
+{
+ (*addr) = cpu_to_le32(val);
+}
+
+static inline u32 cbd_read(__le32 *addr)
+{
+ return le32_to_cpu(*addr);
+}
+
+/*----------------------------------------------------------------------------*/
+/* Doorbell Register Helper Operations */
+/*----------------------------------------------------------------------------*/
+
+static bool wqt_remote_is_ready(struct wqt_dev *priv)
+{
+ /* If the doorbell mask bit is cleared, then someone is listening */
+ return (IMMR_R32(OMIMR_OFFSET) & 0x8) ? false : true;
+}
+
+static inline void wqt_raise_doorbell(struct wqt_dev *priv, u32 dbell)
+{
+ IMMR_W32(ODR_OFFSET, dbell);
+}
+
+/*----------------------------------------------------------------------------*/
+/* Message Sending and Processing Operations */
+/*----------------------------------------------------------------------------*/
+
+static irqreturn_t wqt_interrupt(int irq, void *dev_id)
+{
+ struct wqt_dev *priv = dev_id;
+ u32 imisr, idr;
+
+ imisr = IMMR_R32(IMISR_OFFSET);
+ idr = IMMR_R32(IDR_OFFSET);
+
+ if (!(imisr & 0x8))
+ return IRQ_NONE;
+
+ /* Clear all of the interrupt sources, we'll handle them next */
+ IMMR_W32(IDR_OFFSET, idr);
+
+ /* Lock over all of the handlers, so they cannot get called when
+ * the code doesn't expect them to be called */
+ spin_lock(&priv->irq_lock);
+
+ if (idr & UART_RX_READY_DBELL)
+ priv->handlers.uart_rx_ready_handler(priv);
+
+ if (idr & UART_TX_EMPTY_DBELL)
+ priv->handlers.uart_tx_empty_handler(priv);
+
+ if (idr & NET_RX_PACKET_DBELL)
+ priv->handlers.net_rx_packet_handler(priv);
+
+ if (idr & NET_TX_COMPLETE_DBELL)
+ priv->handlers.net_tx_complete_handler(priv);
+
+ if (idr & NET_START_REQ_DBELL)
+ priv->handlers.net_start_req_handler(priv);
+
+ if (idr & NET_START_ACK_DBELL)
+ priv->handlers.net_start_ack_handler(priv);
+
+ if (idr & NET_STOP_REQ_DBELL)
+ priv->handlers.net_stop_req_handler(priv);
+
+ if (idr & NET_STOP_ACK_DBELL)
+ priv->handlers.net_stop_ack_handler(priv);
+
+ if (idr & NET_DISABLE_RX_PACKET_DBELL)
+ priv->net_send_rx_packet_dbell = false;
+
+ if (idr & NET_ENABLE_RX_PACKET_DBELL)
+ priv->net_send_rx_packet_dbell = true;
+
+ spin_unlock(&priv->irq_lock);
+
+ return IRQ_HANDLED;
+}
+
+/* Send a character through the mbox when it becomes available
+ * Blocking, must not be called with any spinlocks held */
+static int do_send_message(struct wqt_dev *priv, const char ch)
+{
+ struct uart_port *port = &priv->port;
+ bool tmp;
+
+ spin_lock_irq(&priv->irq_lock);
+ while (priv->uart_tx_ready != true) {
+ spin_unlock_irq(&priv->irq_lock);
+ wait_event_timeout(priv->uart_tx_wait, priv->uart_tx_ready, HZ);
+
+ spin_lock_irq(&port->lock);
+ tmp = priv->uart_open;
+ spin_unlock_irq(&port->lock);
+
+ if (!tmp)
+ return -EIO;
+
+ spin_lock_irq(&priv->irq_lock);
+ }
+
+ /* Now the transmitter is free, send the message */
+ IMMR_W32(OMR0_OFFSET, ch);
+ wqt_raise_doorbell(priv, UART_RX_READY_DBELL);
+
+ /* Mark the transmitter busy */
+ priv->uart_tx_ready = false;
+ spin_unlock_irq(&priv->irq_lock);
+ return 0;
+}
+
+/* Grab a character out of the uart tx buffer and send it */
+static void uart_tx_work_fn(struct work_struct *work)
+{
+ struct wqt_dev *priv = container_of(work, struct wqt_dev, uart_tx_work);
+ struct uart_port *port = &priv->port;
+ struct circ_buf *xmit = &port->info->xmit;
+ char ch;
+
+ spin_lock_irq(&port->lock);
+ while (true) {
+
+ /* Check for XON/XOFF (high priority) */
+ if (port->x_char) {
+ ch = port->x_char;
+ port->x_char = 0;
+ spin_unlock_irq(&port->lock);
+
+ if (do_send_message(priv, ch))
+ return;
+
+ spin_lock_irq(&port->lock);
+ continue;
+ }
+
+ /* If we're out of chars or the port is stopped, we're done */
+ if (uart_circ_empty(xmit) || uart_tx_stopped(port)) {
+ wqtuart_stop_tx(port);
+ break;
+ }
+
+ /* Grab the next char out of the buffer and send it */
+ ch = xmit->buf[xmit->tail];
+ xmit->tail = (xmit->tail + 1) & (UART_XMIT_SIZE - 1);
+ spin_unlock_irq(&port->lock);
+
+ if (do_send_message(priv, ch))
+ return;
+
+ spin_lock_irq(&port->lock);
+ }
+
+ if (uart_circ_chars_pending(xmit) < WAKEUP_CHARS)
+ uart_write_wakeup(port);
+
+ if (uart_circ_empty(xmit))
+ wqtuart_stop_tx(port);
+
+ spin_unlock_irq(&port->lock);
+}
+
+/*----------------------------------------------------------------------------*/
+/* Interrupt Handlers */
+/*----------------------------------------------------------------------------*/
+
+/* NOTE: All handlers are called with priv->irq_lock held */
+
+static void empty_handler(struct wqt_dev *priv)
+{
+ /* Intentionally left empty */
+}
+
+static void net_start_req_handler(struct wqt_dev *priv)
+{
+ schedule_work(&priv->net_start_work);
+}
+
+static void net_start_ack_handler(struct wqt_dev *priv)
+{
+ complete(&priv->net_start_completion);
+}
+
+static void net_stop_req_handler(struct wqt_dev *priv)
+{
+ schedule_work(&priv->net_stop_work);
+}
+
+static void net_stop_ack_handler(struct wqt_dev *priv)
+{
+ complete(&priv->net_stop_completion);
+}
+
+static void net_tx_complete_handler(struct wqt_dev *priv)
+{
+ tasklet_schedule(&priv->tx_complete_tasklet);
+}
+
+static void net_rx_packet_handler(struct wqt_dev *priv)
+{
+ wqt_raise_doorbell(priv, NET_DISABLE_RX_PACKET_DBELL);
+ netif_rx_schedule(&priv->napi);
+}
+
+static void uart_rx_ready_handler(struct wqt_dev *priv)
+{
+ struct tty_struct *tty = priv->port.info->port.tty;
+ const char ch = IMMR_R32(IMR0_OFFSET) & 0xff;
+
+ /* Write the character to the tty layer */
+ tty_insert_flip_char(tty, ch, TTY_NORMAL);
+ tty_flip_buffer_push(tty);
+
+ /* Let the other side know we're ready for more */
+ wqt_raise_doorbell(priv, UART_TX_EMPTY_DBELL);
+}
+
+static void uart_tx_empty_handler(struct wqt_dev *priv)
+{
+ priv->uart_tx_ready = true;
+ wake_up(&priv->uart_tx_wait);
+}
+
+/*----------------------------------------------------------------------------*/
+/* Interrupt Request / Free Helpers */
+/*----------------------------------------------------------------------------*/
+
+static void do_enable_net_startstop_handlers(struct wqt_dev *priv)
+{
+ spin_lock_irq(&priv->irq_lock);
+ priv->handlers.net_start_req_handler = net_start_req_handler;
+ priv->handlers.net_start_ack_handler = net_start_ack_handler;
+ priv->handlers.net_stop_req_handler = net_stop_req_handler;
+ priv->handlers.net_stop_ack_handler = net_stop_ack_handler;
+ spin_unlock_irq(&priv->irq_lock);
+}
+
+static void do_disable_net_startstop_handlers(struct wqt_dev *priv)
+{
+ spin_lock_irq(&priv->irq_lock);
+ priv->handlers.net_start_req_handler = empty_handler;
+ priv->handlers.net_start_ack_handler = empty_handler;
+ priv->handlers.net_stop_req_handler = empty_handler;
+ priv->handlers.net_stop_ack_handler = empty_handler;
+ spin_unlock_irq(&priv->irq_lock);
+}
+
+static void do_enable_net_rxtx_handlers(struct wqt_dev *priv)
+{
+ spin_lock_irq(&priv->irq_lock);
+ priv->handlers.net_rx_packet_handler = net_rx_packet_handler;
+ priv->handlers.net_tx_complete_handler = net_tx_complete_handler;
+ spin_unlock_irq(&priv->irq_lock);
+}
+
+static void do_disable_net_rxtx_handlers(struct wqt_dev *priv)
+{
+ spin_lock_irq(&priv->irq_lock);
+ priv->handlers.net_rx_packet_handler = empty_handler;
+ priv->handlers.net_tx_complete_handler = empty_handler;
+ spin_unlock_irq(&priv->irq_lock);
+}
+
+static void do_enable_uart_handlers(struct wqt_dev *priv)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&priv->irq_lock, flags);
+ priv->handlers.uart_rx_ready_handler = uart_rx_ready_handler;
+ priv->handlers.uart_tx_empty_handler = uart_tx_empty_handler;
+ spin_unlock_irqrestore(&priv->irq_lock, flags);
+}
+
+static void do_disable_uart_handlers(struct wqt_dev *priv)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&priv->irq_lock, flags);
+ priv->handlers.uart_rx_ready_handler = empty_handler;
+ priv->handlers.uart_tx_empty_handler = empty_handler;
+ spin_unlock_irqrestore(&priv->irq_lock, flags);
+}
+
+static int wqt_request_irq(struct wqt_dev *priv)
+{
+ int ret = 0;
+
+ mutex_lock(&priv->irq_mutex);
+
+ if (priv->interrupt_count > 0)
+ goto out_unlock;
+
+ /* Force all handlers to be disabled before attaching the handler */
+ do_disable_net_startstop_handlers(priv);
+ do_disable_net_rxtx_handlers(priv);
+ do_disable_uart_handlers(priv);
+
+ ret = request_irq(priv->irq,
+ wqt_interrupt,
+ IRQF_SHARED,
+ priv->ndev->name,
+ priv);
+
+ /* Unmask the doorbell interrupt */
+ IMMR_W32(IMIMR_OFFSET, 0x2 | 0x1);
+
+out_unlock:
+ priv->interrupt_count++;
+ mutex_unlock(&priv->irq_mutex);
+
+ return ret;
+}
+
+static void wqt_free_irq(struct wqt_dev *priv)
+{
+ mutex_lock(&priv->irq_mutex);
+ priv->interrupt_count--;
+
+ if (priv->interrupt_count > 0)
+ goto out_unlock;
+
+ /* Disable all interrupts */
+ IMMR_W32(IMIMR_OFFSET, 0x8 | 0x2 | 0x1);
+
+ free_irq(priv->irq, priv);
+
+out_unlock:
+ mutex_unlock(&priv->irq_mutex);
+}
+
+/*----------------------------------------------------------------------------*/
+/* Network Startup and Shutdown Helpers */
+/*----------------------------------------------------------------------------*/
+
+/* NOTE: All helper functions prefixed with "do" must be called only from
+ * process context, with priv->net_mutex held. They are expected to sleep */
+
+static void do_net_start_queues(struct wqt_dev *priv)
+{
+ if (priv->net_state == NET_STATE_RUNNING)
+ return;
+
+ priv->net_send_rx_packet_dbell = true;
+
+ dev_dbg(priv->dev, "resetting buffer positions\n");
+ priv->cur_rx = priv->rx_base;
+ priv->cur_tx = priv->tx_base;
+ priv->dirty_tx = priv->tx_base;
+ priv->tx_free = PH_RING_SIZE;
+
+ dev_dbg(priv->dev, "enabling NAPI queue\n");
+ napi_enable(&priv->napi);
+
+ dev_dbg(priv->dev, "enabling tx_complete() tasklet\n");
+ tasklet_enable(&priv->tx_complete_tasklet);
+
+ dev_dbg(priv->dev, "enabling TX queue\n");
+ netif_start_queue(priv->ndev);
+
+ dev_dbg(priv->dev, "carrier on!\n");
+ netif_carrier_on(priv->ndev);
+
+ /* Enable the RX_PACKET and TX_COMPLETE interrupt handlers */
+ do_enable_net_rxtx_handlers(priv);
+
+ priv->net_state = NET_STATE_RUNNING;
+}
+
+static void do_net_stop_queues(struct wqt_dev *priv)
+{
+ if (priv->net_state == NET_STATE_STOPPED)
+ return;
+
+ /* Disable the RX_PACKET and TX_COMPLETE interrupt handlers */
+ do_disable_net_rxtx_handlers(priv);
+
+ dev_dbg(priv->dev, "disabling NAPI queue\n");
+ napi_disable(&priv->napi);
+
+ dev_dbg(priv->dev, "disabling tx_complete() tasklet\n");
+ tasklet_disable(&priv->tx_complete_tasklet);
+
+ dev_dbg(priv->dev, "disabling TX queue\n");
+ netif_tx_disable(priv->ndev);
+
+ dev_dbg(priv->dev, "carrier off!\n");
+ netif_carrier_off(priv->ndev);
+
+ priv->net_state = NET_STATE_STOPPED;
+}
+
+/* Called when we get a request to start our queues and acknowledge */
+static void wqtnet_start_work_fn(struct work_struct *work)
+{
+ struct wqt_dev *priv = container_of(work, struct wqt_dev,
+ net_start_work);
+
+ mutex_lock(&priv->net_mutex);
+
+ do_net_start_queues(priv);
+ wqt_raise_doorbell(priv, NET_START_ACK_DBELL);
+
+ mutex_unlock(&priv->net_mutex);
+}
+
+/* Called when we get a request to stop our queues and acknowledge */
+static void wqtnet_stop_work_fn(struct work_struct *work)
+{
+ struct wqt_dev *priv = container_of(work, struct wqt_dev,
+ net_stop_work);
+
+ mutex_lock(&priv->net_mutex);
+
+ do_net_stop_queues(priv);
+ wqt_raise_doorbell(priv, NET_STOP_ACK_DBELL);
+
+ mutex_unlock(&priv->net_mutex);
+}
+
+/*----------------------------------------------------------------------------*/
+/* DMA Operation Helpers */
+/*----------------------------------------------------------------------------*/
+
+/* Setup a static 1GB window starting at PCI address 0x0
+ *
+ * This means that all DMA must be within the first 1GB of the other side's
+ * memory, which shouldn't be a problem
+ */
+static int wqtdma_setup_outbound_window(struct wqt_dev *priv)
+{
+ IMMR_W32BE(LAWAR0_OFFSET, LAWAR0_ENABLE | 0x1d);
+ IMMR_W32BE(POCMR0_OFFSET, POCMR0_ENABLE | 0xc0000);
+ IMMR_W32BE(POTAR0_OFFSET, 0x0);
+
+ return 0;
+}
+
+static enum dma_state_client wqt_dma_event(struct dma_client *client,
+ struct dma_chan *chan,
+ enum dma_state state)
+{
+ struct wqt_dev *priv = container_of(client, struct wqt_dev, client);
+ struct device *chandev = &chan->dev;
+ enum dma_state_client ack = DMA_NAK;
+
+ switch (state) {
+ case DMA_RESOURCE_AVAILABLE:
+ if (chan == priv->rx_chan || chan == priv->tx_chan) {
+ ack = DMA_DUP;
+ break;
+ }
+
+ if (!priv->rx_chan) {
+ priv->rx_chan = chan;
+ dev_info(priv->dev, "RX DMA: %s\n", dev_name(chandev));
+ ack = DMA_ACK;
+ break;
+ }
+
+ if (!priv->tx_chan) {
+ priv->tx_chan = chan;
+ dev_info(priv->dev, "TX DMA: %s\n", dev_name(chandev));
+ ack = DMA_ACK;
+ break;
+ }
+
+ /* Both channels are already allocated */
+ ack = DMA_NAK;
+ break;
+
+ case DMA_RESOURCE_REMOVED:
+ if (chan == priv->rx_chan)
+ priv->rx_chan = NULL;
+
+ if (chan == priv->tx_chan)
+ priv->tx_chan = NULL;
+
+ ack = DMA_ACK;
+ break;
+
+ default:
+ dev_dbg(priv->dev, "unhandled DMA event %u (%s)\n",
+ state, chan->dev.bus_id);
+ break;
+ }
+
+ return ack;
+}
+
+/* NOTE: this function does not automatically unmap the src and dst */
+static dma_cookie_t dma_async_memcpy_raw_to_raw(struct dma_chan *chan,
+ dma_addr_t dst,
+ dma_addr_t src,
+ size_t len)
+{
+ struct dma_device *dev = chan->device;
+ struct dma_async_tx_descriptor *tx;
+ enum dma_ctrl_flags flags;
+ dma_cookie_t cookie;
+ int cpu;
+
+ flags = DMA_COMPL_SKIP_SRC_UNMAP | DMA_COMPL_SKIP_DEST_UNMAP;
+ tx = dev->device_prep_dma_memcpy(chan, dst, src, len, flags);
+ if (!tx)
+ return -ENOMEM;
+
+ tx->callback = NULL;
+ cookie = tx->tx_submit(tx);
+
+ cpu = get_cpu();
+ per_cpu_ptr(chan->local, cpu)->bytes_transferred += len;
+ per_cpu_ptr(chan->local, cpu)->memcpy_count++;
+ put_cpu();
+
+ return cookie;
+}
+
+/*----------------------------------------------------------------------------*/
+/* Network Device Operations */
+/*----------------------------------------------------------------------------*/
+
+static int wqt_open(struct net_device *dev)
+{
+ struct wqt_dev *priv = netdev_priv(dev);
+ int ret;
+
+ /* Pretend the cable is unplugged until we are up and running */
+ netif_carrier_off(dev);
+
+ mutex_lock(&priv->net_mutex);
+
+ ret = wqt_request_irq(priv);
+ if (ret)
+ goto out_unlock;
+
+ /* Enable only the network start/stop interrupts */
+ do_enable_net_startstop_handlers(priv);
+
+ /* Check if the other side is running. If it is not, it will
+ * start us when it comes up */
+ if (!wqt_remote_is_ready(priv)) {
+ ret = 0;
+ goto out_unlock;
+ }
+
+ /* Begin the startup sequence, giving 5 seconds for a reply */
+ wqt_raise_doorbell(priv, NET_START_REQ_DBELL);
+ ret = wait_for_completion_timeout(&priv->net_start_completion, 5*HZ);
+
+ /* The startup sequence timed out, therefore the other side is down.
+ * We'll just leave our interrupt handler hooked up and ready */
+ if (!ret) {
+ dev_warn(priv->dev, "startup sequence timed out\n");
+ ret = 0;
+ goto out_unlock;
+ }
+
+ do_net_start_queues(priv);
+ ret = 0;
+
+out_unlock:
+ mutex_unlock(&priv->net_mutex);
+ return ret;
+}
+
+static int wqt_stop(struct net_device *dev)
+{
+ struct wqt_dev *priv = netdev_priv(dev);
+ int ret;
+
+ mutex_lock(&priv->net_mutex);
+
+ do_net_stop_queues(priv);
+
+ wqt_raise_doorbell(priv, NET_STOP_REQ_DBELL);
+ ret = wait_for_completion_timeout(&priv->net_stop_completion, 5*HZ);
+
+ /* The shutdown sequence timed out, therefore the other side is down
+ * and we shouldn't be messing with the registers */
+ if (!ret)
+ dev_warn(priv->dev, "shutdown sequence timed out\n");
+
+ do_disable_net_startstop_handlers(priv);
+ wqt_free_irq(priv);
+
+ mutex_unlock(&priv->net_mutex);
+ return 0;
+}
+
+static int wqt_change_mtu(struct net_device *dev, int new_mtu)
+{
+ if ((new_mtu < 68) || (new_mtu > PH_MAX_MTU))
+ return -EINVAL;
+
+ dev->mtu = new_mtu;
+ return 0;
+}
+
+static int wqt_hard_start_xmit(struct sk_buff *skb, struct net_device *dev)
+{
+ struct wqt_dev *priv = netdev_priv(dev);
+ struct circ_buf_desc *bdp;
+ dma_addr_t dma_src, dma_dst;
+ dma_cookie_t cookie;
+ enum dma_status status;
+ int dirty_idx;
+
+ spin_lock_bh(&priv->net_lock);
+
+ bdp = priv->cur_tx;
+ dirty_idx = bdp - priv->tx_base;
+
+ /* This should not happen, the queue should be stopped */
+ if (priv->tx_free == 0 || cbd_read(&bdp->sc) != BD_MEM_READY) {
+ dev_warn(priv->dev, "TX queue not stopped!\n");
+ netif_stop_queue(dev);
+ goto out_unlock;
+ }
+
+ if (skb_dma_map(priv->dev, skb, DMA_TO_DEVICE)) {
+ dev_warn(priv->dev, "DMA mapping error\n");
+ goto out_unlock;
+ }
+
+ dma_dst = 0x80000000 + cbd_read(&bdp->addr);
+ dma_src = skb_shinfo(skb)->dma_maps[0];
+ cookie = dma_async_memcpy_raw_to_raw(priv->tx_chan,
+ dma_dst,
+ dma_src,
+ skb->len);
+ if (dma_submit_error(cookie)) {
+ dev_warn(priv->dev, "DMA submit error\n");
+ goto out_unmap;
+ }
+
+ status = dma_sync_wait(priv->tx_chan, cookie);
+ if (status == DMA_ERROR) {
+ dev_warn(priv->dev, "DMA error\n");
+ goto out_unmap;
+ }
+
+ /* Update the buffer descriptor with the packet information */
+ cbd_write(&bdp->len, skb->len);
+ wmb();
+ cbd_write(&bdp->sc, BD_MEM_DIRTY);
+
+ /* We're done with the skb, update stats and free it */
+ dev->stats.tx_bytes += skb->len;
+ dev->stats.tx_packets++;
+ skb_dma_unmap(priv->dev, skb, DMA_TO_DEVICE);
+ dev_kfree_skb_irq(skb);
+
+ if (dirty_idx == PH_RING_SIZE - 1)
+ bdp = priv->tx_base;
+ else
+ bdp++;
+
+ priv->cur_tx = bdp;
+ priv->tx_free--;
+ dev->trans_start = jiffies;
+
+ if (priv->tx_free == 0)
+ netif_stop_queue(dev);
+
+ if (priv->net_send_rx_packet_dbell)
+ wqt_raise_doorbell(priv, NET_RX_PACKET_DBELL);
+
+ spin_unlock_bh(&priv->net_lock);
+ return NETDEV_TX_OK;
+
+out_unmap:
+ skb_dma_unmap(priv->dev, skb, DMA_TO_DEVICE);
+out_unlock:
+ spin_unlock_bh(&priv->net_lock);
+ return NETDEV_TX_BUSY;
+}
+
+static void wqt_tx_timeout(struct net_device *dev)
+{
+ struct wqt_dev *priv = netdev_priv(dev);
+
+ dev->stats.tx_errors++;
+ wqt_raise_doorbell(priv, NET_RX_PACKET_DBELL);
+}
+
+static void wqt_tx_complete(unsigned long data)
+{
+ struct net_device *dev = (struct net_device *)data;
+ struct wqt_dev *priv = netdev_priv(dev);
+ struct circ_buf_desc *bdp;
+ int do_wake, dirty_idx;
+
+ spin_lock_bh(&priv->net_lock);
+
+ bdp = priv->dirty_tx;
+ do_wake = 0;
+
+ while (cbd_read(&bdp->sc) == BD_MEM_FREE) {
+ dirty_idx = bdp - priv->tx_base;
+
+ /* Mark the BDP as ready */
+ cbd_write(&bdp->sc, BD_MEM_READY);
+ wmb();
+
+ /* Update the bdp */
+ if (dirty_idx == PH_RING_SIZE - 1)
+ bdp = priv->tx_base;
+ else
+ bdp++;
+
+ if (!priv->tx_free++)
+ do_wake = 1;
+ }
+
+ priv->dirty_tx = bdp;
+
+ spin_unlock_bh(&priv->net_lock);
+
+ if (do_wake)
+ netif_wake_queue(dev);
+}
+
+static int wqt_rx_napi(struct napi_struct *napi, int budget)
+{
+ struct wqt_dev *priv = container_of(napi, struct wqt_dev, napi);
+ struct net_device *dev = priv->ndev;
+ int dirty_idx;
+ int received = 0;
+ u32 sc, len, addr;
+ struct sk_buff *skb;
+ dma_cookie_t cookie;
+ enum dma_status status;
+ struct circ_buf_desc *bdp;
+ dma_addr_t dma_src, dma_dst;
+
+ bdp = priv->cur_rx;
+
+ while (received < budget) {
+ dirty_idx = bdp - priv->rx_base;
+
+ sc = cbd_read(&bdp->sc);
+ len = cbd_read(&bdp->len);
+ addr = cbd_read(&bdp->addr);
+
+ /* Check if we're out of packets to process */
+ if (sc != BD_MEM_DIRTY)
+ break;
+
+ /* Allocate a packet for the data */
+ skb = dev_alloc_skb(len + NET_IP_ALIGN);
+ if (skb == NULL) {
+ dev->stats.rx_dropped++;
+ goto out_err;
+ }
+
+ skb_reserve(skb, NET_IP_ALIGN);
+
+ if (skb_dma_map(priv->dev, skb, DMA_FROM_DEVICE)) {
+ dev_warn(priv->dev, "DMA mapping error\n");
+ dev_kfree_skb_irq(skb);
+ dev->stats.rx_dropped++;
+ goto out_err;
+ }
+
+ dma_dst = skb_shinfo(skb)->dma_maps[0];
+ dma_src = 0x80000000 + addr;
+
+ cookie = dma_async_memcpy_raw_to_raw(priv->rx_chan,
+ dma_dst,
+ dma_src,
+ len);
+ if (dma_submit_error(cookie)) {
+ dev_warn(priv->dev, "DMA submit error\n");
+ skb_dma_unmap(priv->dev, skb, DMA_FROM_DEVICE);
+ dev_kfree_skb_irq(skb);
+ dev->stats.rx_dropped++;
+ goto out_err;
+ }
+
+ status = dma_sync_wait(priv->rx_chan, cookie);
+ if (status == DMA_ERROR) {
+ dev_warn(priv->dev, "DMA error\n");
+ skb_dma_unmap(priv->dev, skb, DMA_FROM_DEVICE);
+ dev_kfree_skb_irq(skb);
+ dev->stats.rx_dropped++;
+ goto out_err;
+ }
+
+ /* Unmap and push the skb into the network stack */
+ skb_dma_unmap(priv->dev, skb, DMA_FROM_DEVICE);
+ skb_put(skb, len);
+ skb->protocol = eth_type_trans(skb, dev);
+ skb->ip_summed = CHECKSUM_UNNECESSARY;
+ dev->stats.rx_bytes += len;
+ dev->stats.rx_packets++;
+
+ netif_receive_skb(skb);
+ received++;
+
+out_err:
+ cbd_write(&bdp->sc, BD_MEM_FREE);
+ wmb();
+
+ if (dirty_idx == PH_RING_SIZE - 1)
+ bdp = priv->rx_base;
+ else
+ bdp++;
+ }
+
+ priv->cur_rx = bdp;
+
+ /* We have processed all packets that the adapter had, but it
+ * was less than our budget, stop polling */
+ if (received < budget) {
+ netif_rx_complete(napi);
+ wqt_raise_doorbell(priv, NET_ENABLE_RX_PACKET_DBELL);
+ }
+
+ wqt_raise_doorbell(priv, NET_TX_COMPLETE_DBELL);
+
+ return received;
+}
+
+static const struct net_device_ops wqt_net_ops = {
+ .ndo_open = wqt_open,
+ .ndo_stop = wqt_stop,
+ .ndo_change_mtu = wqt_change_mtu,
+ .ndo_start_xmit = wqt_hard_start_xmit,
+ .ndo_tx_timeout = wqt_tx_timeout,
+};
+
+/*----------------------------------------------------------------------------*/
+/* UART Device Operations */
+/*----------------------------------------------------------------------------*/
+
+static unsigned int wqtuart_tx_empty(struct uart_port *port)
+{
+ return TIOCSER_TEMT;
+}
+
+static void wqtuart_set_mctrl(struct uart_port *port, unsigned int mctrl)
+{
+}
+
+static unsigned int wqtuart_get_mctrl(struct uart_port *port)
+{
+ return TIOCM_CAR | TIOCM_DSR | TIOCM_CTS;
+}
+
+static void wqtuart_stop_tx(struct uart_port *port)
+{
+}
+
+static void wqtuart_start_tx(struct uart_port *port)
+{
+ struct wqt_dev *priv = container_of(port, struct wqt_dev, port);
+
+ queue_work(priv->wq, &priv->uart_tx_work);
+}
+
+static void wqtuart_stop_rx(struct uart_port *port)
+{
+ struct wqt_dev *priv = container_of(port, struct wqt_dev, port);
+
+ do_disable_uart_handlers(priv);
+}
+
+static void wqtuart_enable_ms(struct uart_port *port)
+{
+}
+
+static void wqtuart_break_ctl(struct uart_port *port, int break_state)
+{
+}
+
+static int wqtuart_startup(struct uart_port *port)
+{
+ struct wqt_dev *priv = container_of(port, struct wqt_dev, port);
+ int ret;
+
+ ret = wqt_request_irq(priv);
+ if (ret)
+ return ret;
+
+ do_enable_uart_handlers(priv);
+
+ /* Mark the transmitter and receiver ready */
+ priv->uart_tx_ready = true;
+
+ /* Let the other side know that we are ready to receive chars now */
+ wqt_raise_doorbell(priv, UART_TX_EMPTY_DBELL);
+ priv->uart_open = true;
+ return 0;
+}
+
+static void wqtuart_shutdown(struct uart_port *port)
+{
+ struct wqt_dev *priv = container_of(port, struct wqt_dev, port);
+
+ wqt_free_irq(priv);
+
+ /* Make sure the uart_tx_work_fn() exits cleanly */
+ priv->uart_open = false;
+ wake_up(&priv->uart_tx_wait);
+}
+
+static void wqtuart_set_termios(struct uart_port *port,
+ struct ktermios *termios,
+ struct ktermios *old)
+{
+}
+
+static const char *wqtuart_type(struct uart_port *port)
+{
+ return "WQTUART";
+}
+
+static int wqtuart_request_port(struct uart_port *port)
+{
+ return 0;
+}
+
+static void wqtuart_config_port(struct uart_port *port, int flags)
+{
+}
+
+static void wqtuart_release_port(struct uart_port *port)
+{
+}
+
+static int wqtuart_verify_port(struct uart_port *port,
+ struct serial_struct *ser)
+{
+ return 0;
+}
+
+static struct uart_ops wqtuart_ops = {
+ .tx_empty = wqtuart_tx_empty,
+ .set_mctrl = wqtuart_set_mctrl,
+ .get_mctrl = wqtuart_get_mctrl,
+ .stop_tx = wqtuart_stop_tx,
+ .start_tx = wqtuart_start_tx,
+ .stop_rx = wqtuart_stop_rx,
+ .enable_ms = wqtuart_enable_ms,
+ .break_ctl = wqtuart_break_ctl,
+ .startup = wqtuart_startup,
+ .shutdown = wqtuart_shutdown,
+ .set_termios = wqtuart_set_termios,
+ .type = wqtuart_type,
+ .release_port = wqtuart_release_port,
+ .request_port = wqtuart_request_port,
+ .config_port = wqtuart_config_port,
+ .verify_port = wqtuart_verify_port,
+};
+
+static struct uart_driver wqtuart_driver = {
+ .owner = THIS_MODULE,
+ .driver_name = driver_name,
+ .dev_name = "ttyPCI",
+ .major = 240,
+ .minor = 0,
+ .nr = 1,
+};
+
+/*----------------------------------------------------------------------------*/
+/* Network Registers */
+/*----------------------------------------------------------------------------*/
+
+static void wqt_free_netregs(struct wqt_dev *priv)
+{
+ BUG_ON(priv->netregs == NULL);
+ BUG_ON(priv->netregs_addr == 0x0);
+
+ dma_free_coherent(priv->dev,
+ PAGE_SIZE,
+ priv->netregs,
+ priv->netregs_addr);
+
+ priv->netregs = NULL;
+ priv->netregs_addr = 0x0;
+}
+
+static int wqt_init_netregs(struct wqt_dev *priv)
+{
+ u32 val;
+
+ BUG_ON(priv->netregs != NULL);
+ BUG_ON(priv->netregs_addr != 0x0);
+
+ /* Check the PCI Inbound Window Attributes Register 0 for a 4k window
+ * This is PCI BAR1, and will be used as network device registers */
+ val = IMMR_R32BE(PIWAR0_OFFSET);
+ val = val & (PIWAR0_ENABLED | PIWAR0_IWS_4K);
+ if (val != (PIWAR0_ENABLED | PIWAR0_IWS_4K)) {
+ dev_dbg(priv->dev, "PIWAR0 set up incorrectly\n");
+ return -ENODEV;
+ }
+
+ priv->netregs = dma_alloc_coherent(priv->dev,
+ PAGE_SIZE,
+ &priv->netregs_addr,
+ GFP_KERNEL);
+ if (!priv->netregs) {
+ dev_dbg(priv->dev, "Unable to allocate netregs\n");
+ return -ENOMEM;
+ }
+
+ /* Write the page address into the address register */
+ IMMR_W32BE(PITAR0_OFFSET, priv->netregs_addr >> 12);
+ return 0;
+}
+
+/*----------------------------------------------------------------------------*/
+/* OpenFirmware Device Subsystem */
+/*----------------------------------------------------------------------------*/
+
+static int wqt_probe(struct of_device *op, const struct of_device_id *match)
+{
+ struct net_device *ndev;
+ struct wqt_dev *priv;
+ int ret;
+
+ ndev = alloc_etherdev(sizeof(*priv));
+ if (!ndev) {
+ ret = -ENOMEM;
+ goto out_alloc_ndev;
+ }
+
+ dev_set_drvdata(&op->dev, ndev);
+ priv = netdev_priv(ndev);
+ priv->dev = &op->dev;
+ priv->ndev = ndev;
+
+ spin_lock_init(&priv->irq_lock);
+ mutex_init(&priv->irq_mutex);
+
+ /* Hardware Initialization */
+ priv->irq = irq_of_parse_and_map(op->node, 0);
+ priv->immr = ioremap(get_immrbase(), 0x100000);
+ if (!priv->immr) {
+ ret = -ENOMEM;
+ goto out_ioremap_immr;
+ }
+
+ ret = wqt_init_netregs(priv);
+ if (ret)
+ goto out_init_netregs;
+
+ /* NOTE: Yes, this is correct. Everything was written as if this
+ * NOTE: side *is* a network card. So the place the card is
+ * NOTE: receiving from is the other side's TX buffers */
+ priv->rx_base = priv->netregs + PCINET_TXBD_BASE;
+ priv->tx_base = priv->netregs + PCINET_RXBD_BASE;
+
+ /* DMA Client */
+ wqtdma_setup_outbound_window(priv);
+ priv->client.event_callback = wqt_dma_event;
+ dma_cap_set(DMA_MEMCPY, priv->client.cap_mask);
+ dma_async_client_register(&priv->client);
+ dma_async_client_chan_request(&priv->client);
+
+ /* Initialize private data */
+ priv->wq = create_singlethread_workqueue(driver_name);
+ if (!priv->wq) {
+ ret = -ENOMEM;
+ goto out_create_workqueue;
+ }
+
+ INIT_WORK(&priv->uart_tx_work, uart_tx_work_fn);
+ init_waitqueue_head(&priv->uart_tx_wait);
+ priv->uart_tx_ready = true;
+
+ tasklet_init(&priv->tx_complete_tasklet, wqt_tx_complete,
+ (unsigned long)ndev);
+ tasklet_disable(&priv->tx_complete_tasklet);
+ spin_lock_init(&priv->net_lock);
+
+ mutex_init(&priv->net_mutex);
+ priv->net_state = NET_STATE_STOPPED;
+ INIT_WORK(&priv->net_start_work, wqtnet_start_work_fn);
+ INIT_WORK(&priv->net_stop_work, wqtnet_stop_work_fn);
+ init_completion(&priv->net_start_completion);
+ init_completion(&priv->net_stop_completion);
+
+ /* Disable all interrupts */
+ IMMR_W32(IMIMR_OFFSET, 0x8 | 0x2 | 0x1);
+
+ /* Network Device */
+ random_ether_addr(ndev->dev_addr);
+
+ ndev->netdev_ops = &wqt_net_ops;
+ ndev->watchdog_timeo = HZ / 4;
+ ndev->flags &= ~IFF_MULTICAST; /* No multicast support */
+ ndev->mtu = PH_MAX_MTU;
+ netif_napi_add(ndev, &priv->napi, wqt_rx_napi, PH_RING_SIZE);
+
+ ret = register_netdev(ndev);
+ if (ret)
+ goto out_register_netdev;
+
+ /* UART Device */
+ priv->port.ops = &wqtuart_ops;
+ priv->port.type = PORT_16550A;
+ priv->port.dev = &op->dev;
+ priv->port.line = 0;
+ spin_lock_init(&priv->port.lock);
+
+ ret = uart_add_one_port(&wqtuart_driver, &priv->port);
+ if (ret)
+ goto out_add_uart_port;
+
+ dev_info(priv->dev, "using ethernet device %s\n", ndev->name);
+ dev_info(priv->dev, "using serial device %s%d\n",
+ wqtuart_driver.dev_name, priv->port.line);
+ return 0;
+
+out_add_uart_port:
+ unregister_netdev(ndev);
+out_register_netdev:
+ destroy_workqueue(priv->wq);
+out_create_workqueue:
+ wqt_free_netregs(priv);
+out_init_netregs:
+ iounmap(priv->immr);
+out_ioremap_immr:
+ free_netdev(ndev);
+out_alloc_ndev:
+ return ret;
+}
+
+static int wqt_remove(struct of_device *op)
+{
+ struct net_device *ndev = dev_get_drvdata(&op->dev);
+ struct wqt_dev *priv = netdev_priv(ndev);
+
+ uart_remove_one_port(&wqtuart_driver, &priv->port);
+ unregister_netdev(priv->ndev);
+
+ flush_workqueue(priv->wq);
+ destroy_workqueue(priv->wq);
+
+ /* Disable all interrupts */
+ IMMR_W32(IMIMR_OFFSET, 0x8 | 0x2 | 0x1);
+
+ wqt_free_netregs(priv);
+
+ dma_async_client_unregister(&priv->client);
+
+ iounmap(priv->immr);
+
+ free_netdev(ndev);
+
+ return 0;
+}
+
+static struct of_device_id wqt_match[] = {
+ { .compatible = "fsl,mpc8349-mu", },
+ {},
+};
+
+static struct of_platform_driver wqt_of_driver = {
+ .owner = THIS_MODULE,
+ .name = driver_name,
+ .match_table = wqt_match,
+ .probe = wqt_probe,
+ .remove = wqt_remove,
+};
+
+/*----------------------------------------------------------------------------*/
+/* DMA Client Infrastructure */
+/*----------------------------------------------------------------------------*/
+
+/*----------------------------------------------------------------------------*/
+/* Module Init / Exit */
+/*----------------------------------------------------------------------------*/
+
+static int __init wqt_init(void)
+{
+ int ret;
+
+ ret = uart_register_driver(&wqtuart_driver);
+ if (ret)
+ goto out_uart_register_driver;
+
+ ret = of_register_platform_driver(&wqt_of_driver);
+ if (ret)
+ goto out_of_register_platform_driver;
+
+ return 0;
+
+out_of_register_platform_driver:
+ uart_unregister_driver(&wqtuart_driver);
+out_uart_register_driver:
+ return ret;
+}
+
+static void __exit wqt_exit(void)
+{
+ of_unregister_platform_driver(&wqt_of_driver);
+ uart_unregister_driver(&wqtuart_driver);
+}
+
+MODULE_AUTHOR("Ira W. Snyder <[email protected]>");
+MODULE_DESCRIPTION("PCINet/PCISerial Driver for MPC8349EMDS");
+MODULE_LICENSE("GPL");
+
+module_init(wqt_init);
+module_exit(wqt_exit);
diff --git a/drivers/net/pcinet_host.c b/drivers/net/pcinet_host.c
new file mode 100644
index 0000000..3b5c38a
--- /dev/null
+++ b/drivers/net/pcinet_host.c
@@ -0,0 +1,1318 @@
+/*
+ * PCINet and PCISerial Driver for Freescale MPC8349EMDS (Host side)
+ *
+ * Copyright (c) 2008 Ira W. Snyder <[email protected]>
+ *
+ * Heavily inspired by the drivers/net/fs_enet driver
+ *
+ * This file is licensed under the terms of the GNU General Public License
+ * version 2. This program is licensed "as is" without any warranty of any
+ * kind, whether express or implied.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/sched.h>
+#include <linux/wait.h>
+#include <linux/interrupt.h>
+#include <linux/irqreturn.h>
+#include <linux/pci.h>
+#include <linux/serial.h>
+#include <linux/serial_core.h>
+#include <linux/etherdevice.h>
+#include <linux/mutex.h>
+
+#include "pcinet.h"
+#include "pcinet_hw.h"
+
+/* IMMR Accessor Helpers */
+#define IMMR_R32(_off) ioread32(priv->immr+(_off))
+#define IMMR_W32(_off, _val) iowrite32((_val), priv->immr+(_off))
+#define IMMR_R32BE(_off) ioread32be(priv->immr+(_off))
+#define IMMR_W32BE(_off, _val) iowrite32be((_val), priv->immr+(_off))
+
+static const char driver_name[] = "wqt";
+
+static void wqtuart_stop_tx(struct uart_port *port);
+
+struct wqt_dev;
+typedef void (*wqt_irqhandler_t)(struct wqt_dev *);
+
+struct wqt_irqhandlers {
+ wqt_irqhandler_t net_start_req_handler;
+ wqt_irqhandler_t net_start_ack_handler;
+ wqt_irqhandler_t net_stop_req_handler;
+ wqt_irqhandler_t net_stop_ack_handler;
+ wqt_irqhandler_t net_rx_packet_handler;
+ wqt_irqhandler_t net_tx_complete_handler;
+ wqt_irqhandler_t uart_rx_ready_handler;
+ wqt_irqhandler_t uart_tx_empty_handler;
+};
+
+struct wqt_dev {
+ /*--------------------------------------------------------------------*/
+ /* PCI Infrastructure */
+ /*--------------------------------------------------------------------*/
+ struct pci_dev *pdev;
+ struct device *dev;
+ void __iomem *immr;
+
+ struct mutex irq_mutex;
+ int interrupt_count;
+
+ spinlock_t irq_lock;
+ struct wqt_irqhandlers handlers;
+
+ /*--------------------------------------------------------------------*/
+ /* UART Device Infrastructure */
+ /*--------------------------------------------------------------------*/
+ struct uart_port port;
+ bool uart_open;
+
+ struct workqueue_struct *wq;
+ struct work_struct uart_tx_work;
+ wait_queue_head_t uart_tx_wait; /* sleep for uart_tx_ready */
+ bool uart_tx_ready; /* transmitter state */
+
+ /*--------------------------------------------------------------------*/
+ /* Ethernet Device Infrastructure */
+ /*--------------------------------------------------------------------*/
+ struct net_device *ndev;
+ void __iomem *netregs;
+
+ /* Outstanding SKB */
+ struct sk_buff *rx_skbs[PH_RING_SIZE];
+ struct sk_buff *tx_skbs[PH_RING_SIZE];
+
+ /* Circular Buffer Descriptor base */
+ struct circ_buf_desc __iomem *rx_base;
+ struct circ_buf_desc __iomem *tx_base;
+
+ /* Current SKB index */
+ struct circ_buf_desc __iomem *cur_rx;
+ struct circ_buf_desc __iomem *cur_tx;
+ struct circ_buf_desc __iomem *dirty_tx;
+ int tx_free;
+
+ struct tasklet_struct tx_complete_tasklet;
+ spinlock_t net_lock;
+
+ struct mutex net_mutex;
+ int net_state;
+ struct work_struct net_start_work;
+ struct work_struct net_stop_work;
+ struct completion net_start_completion;
+ struct completion net_stop_completion;
+ struct napi_struct napi;
+
+ bool net_send_rx_packet_dbell;
+};
+
+/*----------------------------------------------------------------------------*/
+/* Buffer Descriptor Accessor Helpers */
+/*----------------------------------------------------------------------------*/
+
+static inline void cbd_write(void __iomem *addr, u32 val)
+{
+ iowrite32(val, addr);
+}
+
+static inline u32 cbd_read(void __iomem *addr)
+{
+ return ioread32(addr);
+}
+
+/*----------------------------------------------------------------------------*/
+/* Doorbell Register Helper Operations */
+/*----------------------------------------------------------------------------*/
+
+static bool wqt_remote_is_ready(struct wqt_dev *priv)
+{
+ /* If the doorbell mask bit is cleared, then someone is listening */
+ return (IMMR_R32(IMIMR_OFFSET) & 0x8) ? false : true;
+}
+
+/* Set a doorbell bit */
+static inline void wqt_raise_doorbell(struct wqt_dev *priv, u32 dbell)
+{
+ IMMR_W32(IDR_OFFSET, dbell);
+}
+
+/*----------------------------------------------------------------------------*/
+/* Message Sending and Processing Operations */
+/*----------------------------------------------------------------------------*/
+
+static irqreturn_t wqt_interrupt(int irq, void *dev_id)
+{
+ struct wqt_dev *priv = dev_id;
+ u32 omisr, odr;
+
+ omisr = IMMR_R32(OMISR_OFFSET);
+ odr = IMMR_R32(ODR_OFFSET);
+
+ if (!(omisr & 0x8))
+ return IRQ_NONE;
+
+ /* Clear all of the interrupt sources, we'll handle them next */
+ IMMR_W32(ODR_OFFSET, odr);
+
+ /* Lock over all of the handlers, so they cannot get called when
+ * the code doesn't expect them to be called */
+ spin_lock(&priv->irq_lock);
+
+ if (odr & UART_RX_READY_DBELL)
+ priv->handlers.uart_rx_ready_handler(priv);
+
+ if (odr & UART_TX_EMPTY_DBELL)
+ priv->handlers.uart_tx_empty_handler(priv);
+
+ if (odr & NET_RX_PACKET_DBELL)
+ priv->handlers.net_rx_packet_handler(priv);
+
+ if (odr & NET_TX_COMPLETE_DBELL)
+ priv->handlers.net_tx_complete_handler(priv);
+
+ if (odr & NET_START_REQ_DBELL)
+ priv->handlers.net_start_req_handler(priv);
+
+ if (odr & NET_START_ACK_DBELL)
+ priv->handlers.net_start_ack_handler(priv);
+
+ if (odr & NET_STOP_REQ_DBELL)
+ priv->handlers.net_stop_req_handler(priv);
+
+ if (odr & NET_STOP_ACK_DBELL)
+ priv->handlers.net_stop_ack_handler(priv);
+
+ if (odr & NET_DISABLE_RX_PACKET_DBELL)
+ priv->net_send_rx_packet_dbell = false;
+
+ if (odr & NET_ENABLE_RX_PACKET_DBELL)
+ priv->net_send_rx_packet_dbell = true;
+
+ spin_unlock(&priv->irq_lock);
+
+ return IRQ_HANDLED;
+}
+
+/* Send a character through the mbox when it becomes available
+ * Blocking, must not be called with any spinlocks held */
+static int do_send_message(struct wqt_dev *priv, const char ch)
+{
+ struct uart_port *port = &priv->port;
+ bool tmp;
+
+ spin_lock_irq(&priv->irq_lock);
+ while (priv->uart_tx_ready != true) {
+ spin_unlock_irq(&priv->irq_lock);
+ wait_event_timeout(priv->uart_tx_wait, priv->uart_tx_ready, HZ);
+
+ spin_lock_irq(&port->lock);
+ tmp = priv->uart_open;
+ spin_unlock_irq(&port->lock);
+
+ if (!tmp)
+ return -EIO;
+
+ spin_lock_irq(&priv->irq_lock);
+ }
+
+ /* Now the transmitter is free, send the message */
+ IMMR_W32(IMR0_OFFSET, ch);
+ wqt_raise_doorbell(priv, UART_RX_READY_DBELL);
+
+ /* Mark the transmitter busy */
+ priv->uart_tx_ready = false;
+ spin_unlock_irq(&priv->irq_lock);
+ return 0;
+}
+
+/* Grab a character out of the uart tx buffer and send it */
+static void uart_tx_work_fn(struct work_struct *work)
+{
+ struct wqt_dev *priv = container_of(work, struct wqt_dev, uart_tx_work);
+ struct uart_port *port = &priv->port;
+ struct circ_buf *xmit = &port->info->xmit;
+ char ch;
+
+ spin_lock_irq(&port->lock);
+ while (true) {
+
+ /* Check for XON/XOFF (high priority) */
+ if (port->x_char) {
+ ch = port->x_char;
+ port->x_char = 0;
+ spin_unlock_irq(&port->lock);
+
+ if (do_send_message(priv, ch))
+ return;
+
+ spin_lock_irq(&port->lock);
+ continue;
+ }
+
+ /* If we're out of chars or the port is stopped, we're done */
+ if (uart_circ_empty(xmit) || uart_tx_stopped(port)) {
+ wqtuart_stop_tx(port);
+ break;
+ }
+
+ /* Grab the next char out of the buffer and send it */
+ ch = xmit->buf[xmit->tail];
+ xmit->tail = (xmit->tail + 1) & (UART_XMIT_SIZE - 1);
+ spin_unlock_irq(&port->lock);
+
+ if (do_send_message(priv, ch))
+ return;
+
+ spin_lock_irq(&port->lock);
+ }
+
+ if (uart_circ_chars_pending(xmit) < WAKEUP_CHARS)
+ uart_write_wakeup(port);
+
+ if (uart_circ_empty(xmit))
+ wqtuart_stop_tx(port);
+
+ spin_unlock_irq(&port->lock);
+}
+
+/*----------------------------------------------------------------------------*/
+/* Interrupt Handlers */
+/*----------------------------------------------------------------------------*/
+
+/* NOTE: All handlers are called with priv->irq_lock held */
+
+static void empty_handler(struct wqt_dev *priv)
+{
+ /* Intentionally left empty */
+}
+
+static void net_start_req_handler(struct wqt_dev *priv)
+{
+ schedule_work(&priv->net_start_work);
+}
+
+static void net_start_ack_handler(struct wqt_dev *priv)
+{
+ complete(&priv->net_start_completion);
+}
+
+static void net_stop_req_handler(struct wqt_dev *priv)
+{
+ schedule_work(&priv->net_stop_work);
+}
+
+static void net_stop_ack_handler(struct wqt_dev *priv)
+{
+ complete(&priv->net_stop_completion);
+}
+
+static void net_tx_complete_handler(struct wqt_dev *priv)
+{
+ tasklet_schedule(&priv->tx_complete_tasklet);
+}
+
+static void net_rx_packet_handler(struct wqt_dev *priv)
+{
+ wqt_raise_doorbell(priv, NET_DISABLE_RX_PACKET_DBELL);
+ netif_rx_schedule(&priv->napi);
+}
+
+static void uart_rx_ready_handler(struct wqt_dev *priv)
+{
+ struct tty_struct *tty = priv->port.info->port.tty;
+ const char ch = IMMR_R32(OMR0_OFFSET) & 0xff;
+
+ /* Write the character to the tty layer */
+ tty_insert_flip_char(tty, ch, TTY_NORMAL);
+ tty_flip_buffer_push(tty);
+
+ /* Let the other side know we're ready for more */
+ wqt_raise_doorbell(priv, UART_TX_EMPTY_DBELL);
+}
+
+static void uart_tx_empty_handler(struct wqt_dev *priv)
+{
+ priv->uart_tx_ready = true;
+ wake_up(&priv->uart_tx_wait);
+}
+
+/*----------------------------------------------------------------------------*/
+/* Interrupt Request / Free Helpers */
+/*----------------------------------------------------------------------------*/
+
+static void do_enable_net_startstop_handlers(struct wqt_dev *priv)
+{
+ spin_lock_irq(&priv->irq_lock);
+ priv->handlers.net_start_req_handler = net_start_req_handler;
+ priv->handlers.net_start_ack_handler = net_start_ack_handler;
+ priv->handlers.net_stop_req_handler = net_stop_req_handler;
+ priv->handlers.net_stop_ack_handler = net_stop_ack_handler;
+ spin_unlock_irq(&priv->irq_lock);
+}
+
+static void do_disable_net_startstop_handlers(struct wqt_dev *priv)
+{
+ spin_lock_irq(&priv->irq_lock);
+ priv->handlers.net_start_req_handler = empty_handler;
+ priv->handlers.net_start_ack_handler = empty_handler;
+ priv->handlers.net_stop_req_handler = empty_handler;
+ priv->handlers.net_stop_ack_handler = empty_handler;
+ spin_unlock_irq(&priv->irq_lock);
+}
+
+static void do_enable_net_rxtx_handlers(struct wqt_dev *priv)
+{
+ spin_lock_irq(&priv->irq_lock);
+ priv->handlers.net_rx_packet_handler = net_rx_packet_handler;
+ priv->handlers.net_tx_complete_handler = net_tx_complete_handler;
+ spin_unlock_irq(&priv->irq_lock);
+}
+
+static void do_disable_net_rxtx_handlers(struct wqt_dev *priv)
+{
+ spin_lock_irq(&priv->irq_lock);
+ priv->handlers.net_rx_packet_handler = empty_handler;
+ priv->handlers.net_tx_complete_handler = empty_handler;
+ spin_unlock_irq(&priv->irq_lock);
+}
+
+static void do_enable_uart_handlers(struct wqt_dev *priv)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&priv->irq_lock, flags);
+ priv->handlers.uart_rx_ready_handler = uart_rx_ready_handler;
+ priv->handlers.uart_tx_empty_handler = uart_tx_empty_handler;
+ spin_unlock_irqrestore(&priv->irq_lock, flags);
+}
+
+static void do_disable_uart_handlers(struct wqt_dev *priv)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&priv->irq_lock, flags);
+ priv->handlers.uart_rx_ready_handler = empty_handler;
+ priv->handlers.uart_tx_empty_handler = empty_handler;
+ spin_unlock_irqrestore(&priv->irq_lock, flags);
+}
+
+static int wqt_request_irq(struct wqt_dev *priv)
+{
+ int ret = 0;
+
+ mutex_lock(&priv->irq_mutex);
+
+ if (priv->interrupt_count > 0)
+ goto out_unlock;
+
+ /* Force all handlers to be disabled before attaching the handler */
+ do_disable_net_startstop_handlers(priv);
+ do_disable_net_rxtx_handlers(priv);
+ do_disable_uart_handlers(priv);
+
+ /* NOTE: we MUST use pdev->irq here, sometimes it changes AFTER
+ * NOTE: probe() has finished.... */
+ ret = request_irq(priv->pdev->irq,
+ wqt_interrupt,
+ IRQF_SHARED,
+ priv->ndev->name,
+ priv);
+
+ /* Unmask the doorbell interrupt */
+ IMMR_W32(OMIMR_OFFSET, 0x2 | 0x1);
+
+out_unlock:
+ priv->interrupt_count++;
+ mutex_unlock(&priv->irq_mutex);
+
+ return ret;
+}
+
+static void wqt_free_irq(struct wqt_dev *priv)
+{
+ mutex_lock(&priv->irq_mutex);
+ priv->interrupt_count--;
+
+ if (priv->interrupt_count > 0)
+ goto out_unlock;
+
+ /* Disable all interrupts */
+ IMMR_W32(OMIMR_OFFSET, 0x8 | 0x2 | 0x1);
+
+ free_irq(priv->pdev->irq, priv);
+
+out_unlock:
+ mutex_unlock(&priv->irq_mutex);
+}
+
+/*----------------------------------------------------------------------------*/
+/* Network Startup and Shutdown Helpers */
+/*----------------------------------------------------------------------------*/
+
+/* NOTE: All helper functions prefixed with "do" must be called only from
+ * process context, with priv->net_mutex held. They are expected to sleep */
+
+/* NOTE: queues must be stopped before initializing and uninitializing */
+
+static void do_net_initialize_board(struct wqt_dev *priv)
+{
+ int i;
+ struct sk_buff *skb;
+ struct circ_buf_desc __iomem *bdp;
+
+ /* Fill in RX ring */
+ for (i = 0, bdp = priv->rx_base; i < PH_RING_SIZE; bdp++, i++) {
+ skb = priv->rx_skbs[i];
+
+ cbd_write(&bdp->sc, BD_MEM_READY);
+ cbd_write(&bdp->len, PH_MAX_FRSIZE);
+ cbd_write(&bdp->addr, skb_shinfo(skb)->dma_maps[0]);
+ }
+
+ /* Fill in TX ring */
+ for (i = 0, bdp = priv->tx_base; i < PH_RING_SIZE; bdp++, i++) {
+ cbd_write(&bdp->sc, BD_MEM_READY);
+ cbd_write(&bdp->len, 0);
+ cbd_write(&bdp->addr, 0x0);
+ }
+}
+
+static void do_net_uninitialize_board(struct wqt_dev *priv)
+{
+ struct sk_buff *skb;
+ struct circ_buf_desc __iomem *bdp;
+ int i;
+
+ /* Reset TX ring */
+ for (i = 0, bdp = priv->tx_base; i < PH_RING_SIZE; bdp++, i++) {
+ if (priv->tx_skbs[i]) {
+ skb = priv->tx_skbs[i];
+ skb_dma_unmap(priv->dev, skb, DMA_TO_DEVICE);
+ dev_kfree_skb(skb);
+
+ priv->tx_skbs[i] = NULL;
+ }
+
+ cbd_write(&bdp->sc, BD_MEM_READY);
+ cbd_write(&bdp->len, 0);
+ cbd_write(&bdp->addr, 0x0);
+ }
+}
+
+static void do_net_start_queues(struct wqt_dev *priv)
+{
+ if (priv->net_state == NET_STATE_RUNNING)
+ return;
+
+ priv->net_send_rx_packet_dbell = true;
+
+ dev_dbg(priv->dev, "resetting buffer positions\n");
+ priv->cur_rx = priv->rx_base;
+ priv->cur_tx = priv->tx_base;
+ priv->dirty_tx = priv->tx_base;
+ priv->tx_free = PH_RING_SIZE;
+
+ dev_dbg(priv->dev, "enabling NAPI queue\n");
+ napi_enable(&priv->napi);
+
+ dev_dbg(priv->dev, "enabling tx_complete() tasklet\n");
+ tasklet_enable(&priv->tx_complete_tasklet);
+
+ dev_dbg(priv->dev, "enabling TX queue\n");
+ netif_start_queue(priv->ndev);
+
+ dev_dbg(priv->dev, "carrier on!\n");
+ netif_carrier_on(priv->ndev);
+
+ /* Enable the RX_PACKET and TX_COMPLETE interrupt handlers */
+ do_enable_net_rxtx_handlers(priv);
+
+ priv->net_state = NET_STATE_RUNNING;
+}
+
+static void do_net_stop_queues(struct wqt_dev *priv)
+{
+ if (priv->net_state == NET_STATE_STOPPED)
+ return;
+
+ /* Disable the RX_PACKET and TX_COMPLETE interrupt handlers */
+ do_disable_net_rxtx_handlers(priv);
+
+ dev_dbg(priv->dev, "disabling NAPI queue\n");
+ napi_disable(&priv->napi);
+
+ dev_dbg(priv->dev, "disabling tx_complete() tasklet\n");
+ tasklet_disable(&priv->tx_complete_tasklet);
+
+ dev_dbg(priv->dev, "disabling TX queue\n");
+ netif_tx_disable(priv->ndev);
+
+ dev_dbg(priv->dev, "carrier off!\n");
+ netif_carrier_off(priv->ndev);
+
+ priv->net_state = NET_STATE_STOPPED;
+}
+
+/* Called when we get a request to start our queues and acknowledge */
+static void wqtnet_start_work_fn(struct work_struct *work)
+{
+ struct wqt_dev *priv = container_of(work, struct wqt_dev,
+ net_start_work);
+
+ mutex_lock(&priv->net_mutex);
+
+ do_net_initialize_board(priv);
+ do_net_start_queues(priv);
+ wqt_raise_doorbell(priv, NET_START_ACK_DBELL);
+
+ mutex_unlock(&priv->net_mutex);
+}
+
+/* Called when we get a request to stop our queues and acknowledge */
+static void wqtnet_stop_work_fn(struct work_struct *work)
+{
+ struct wqt_dev *priv = container_of(work, struct wqt_dev,
+ net_stop_work);
+
+ mutex_lock(&priv->net_mutex);
+
+ do_net_stop_queues(priv);
+ do_net_uninitialize_board(priv);
+ wqt_raise_doorbell(priv, NET_STOP_ACK_DBELL);
+
+ mutex_unlock(&priv->net_mutex);
+}
+
+/*----------------------------------------------------------------------------*/
+/* SKB Allocation Helpers */
+/*----------------------------------------------------------------------------*/
+
+static void wqt_cleanup_skbs(struct wqt_dev *priv)
+{
+ struct sk_buff *skb;
+ int i;
+
+ /* TX ring */
+ for (i = 0; i < PH_RING_SIZE; ++i) {
+ skb = priv->tx_skbs[i];
+
+ if (skb) {
+ skb_dma_unmap(priv->dev, skb, DMA_TO_DEVICE);
+ dev_kfree_skb(skb);
+ priv->tx_skbs[i] = NULL;
+ }
+ }
+
+ /* RX ring */
+ for (i = 0; i < PH_RING_SIZE; ++i) {
+ skb = priv->rx_skbs[i];
+
+ if (skb) {
+ skb_dma_unmap(priv->dev, skb, DMA_FROM_DEVICE);
+ dev_kfree_skb(skb);
+ priv->rx_skbs[i] = NULL;
+ }
+ }
+}
+
+/*
+ * Allocate and DMA map a new skb to receive a packet
+ *
+ * When the skb is received, it must be unmapped and trimmed
+ * to the appropriate length
+ */
+static struct sk_buff *wqt_alloc_rx_skb(struct wqt_dev *priv)
+{
+ struct net_device *dev = priv->ndev;
+ struct sk_buff *skb;
+
+ skb = netdev_alloc_skb(dev, PH_MAX_FRSIZE + NET_IP_ALIGN);
+ if (unlikely(!skb))
+ return NULL;
+
+ skb_reserve(skb, NET_IP_ALIGN);
+
+ /* We "fill" the skb with data so that the DMA mapping code works
+ * as expected. The skb will be trimmed down to the actual data
+ * size during RX */
+ skb_put(skb, PH_MAX_FRSIZE);
+
+ if (unlikely(skb_dma_map(priv->dev, skb, DMA_FROM_DEVICE))) {
+ dev_kfree_skb_any(skb);
+ return NULL;
+ }
+
+ return skb;
+}
+
+static int wqt_alloc_skbs(struct wqt_dev *priv)
+{
+ struct sk_buff *skb;
+ int i;
+
+ /* RX ring */
+ for (i = 0; i < PH_RING_SIZE; ++i) {
+ /* Paranoia check */
+ BUG_ON(priv->rx_skbs[i] != NULL);
+
+ skb = wqt_alloc_rx_skb(priv);
+ if (skb == NULL)
+ goto out_err;
+
+ priv->rx_skbs[i] = skb;
+ }
+
+ /* TX ring */
+ for (i = 0; i < PH_RING_SIZE; ++i) {
+ /* Paranoia check */
+ BUG_ON(priv->tx_skbs[i] != NULL);
+ }
+
+ /* NOTE: the actual initialization of the board happens
+ * NOTE: in ph_initialize_board(), once the board has
+ * NOTE: requested to be initialized */
+
+ return 0;
+
+out_err:
+ wqt_cleanup_skbs(priv);
+ return -ENOMEM;
+}
+
+/*----------------------------------------------------------------------------*/
+/* Network Device Operations */
+/*----------------------------------------------------------------------------*/
+
+static int wqt_open(struct net_device *dev)
+{
+ struct wqt_dev *priv = netdev_priv(dev);
+ int ret;
+
+ /* Pretend the cable is unplugged until we are up and running */
+ netif_carrier_off(dev);
+
+ /* Check that the other side has set up the network registers */
+ if (!wqt_remote_is_ready(priv)) {
+ dev_err(priv->dev, "no driver installed at other end\n");
+ return -ENOTCONN; /* Transport endpoint not connected */
+ }
+
+ mutex_lock(&priv->net_mutex);
+
+ /* Allocate skbs, initialize the board, start the irq handler */
+ ret = wqt_alloc_skbs(priv);
+ if (ret)
+ goto out_err;
+
+ do_net_initialize_board(priv);
+
+ ret = wqt_request_irq(priv);
+ if (ret)
+ goto out_err;
+
+ /* Enable only the network start/stop interrupts */
+ do_enable_net_startstop_handlers(priv);
+
+ /* Begin the startup sequence, giving 5 seconds for a reply */
+ wqt_raise_doorbell(priv, NET_START_REQ_DBELL);
+ ret = wait_for_completion_timeout(&priv->net_start_completion, 5*HZ);
+
+ /* The startup sequence timed out, therefore the other side is down
+ * and we shouldn't be messing with the registers */
+ if (!ret) {
+ dev_warn(priv->dev, "startup sequence timed out\n");
+ ret = -ENOTCONN;
+ goto out_free_irq;
+ }
+
+ /* Start the network queues */
+ do_net_start_queues(priv);
+
+ mutex_unlock(&priv->net_mutex);
+ return 0;
+
+out_free_irq:
+ wqt_free_irq(priv);
+out_err:
+ wqt_cleanup_skbs(priv);
+ mutex_unlock(&priv->net_mutex);
+ return ret;
+}
+
+static int wqt_stop(struct net_device *dev)
+{
+ struct wqt_dev *priv = netdev_priv(dev);
+ int ret;
+
+ mutex_lock(&priv->net_mutex);
+
+ do_net_stop_queues(priv);
+
+ wqt_raise_doorbell(priv, NET_STOP_REQ_DBELL);
+ ret = wait_for_completion_timeout(&priv->net_stop_completion, 5*HZ);
+
+ /* The shutdown sequence timed out, therefore the other side is down
+ * and we shouldn't be messing with the registers */
+ if (!ret)
+ dev_warn(priv->dev, "shutdown sequence timed out\n");
+
+ do_disable_net_startstop_handlers(priv);
+ wqt_free_irq(priv);
+ do_net_uninitialize_board(priv);
+ wqt_cleanup_skbs(priv);
+
+ mutex_unlock(&priv->net_mutex);
+ return 0;
+}
+
+static int wqt_change_mtu(struct net_device *dev, int new_mtu)
+{
+ if ((new_mtu < 68) || (new_mtu > PH_MAX_MTU))
+ return -EINVAL;
+
+ dev->mtu = new_mtu;
+ return 0;
+}
+
+static int wqt_hard_start_xmit(struct sk_buff *skb, struct net_device *dev)
+{
+ struct wqt_dev *priv = netdev_priv(dev);
+ struct circ_buf_desc __iomem *bdp;
+ int dirty_idx;
+
+ spin_lock_bh(&priv->net_lock);
+
+ bdp = priv->cur_tx;
+ dirty_idx = bdp - priv->tx_base;
+
+ /* This should not happen, the queue should be stopped */
+ if (priv->tx_free == 0 || cbd_read(&bdp->sc) != BD_MEM_READY) {
+ dev_warn(priv->dev, "TX queue not stopped!\n");
+ netif_stop_queue(dev);
+ goto out_unlock;
+ }
+
+ if (skb_dma_map(priv->dev, skb, DMA_TO_DEVICE)) {
+ dev_warn(priv->dev, "DMA mapping error\n");
+ goto out_unlock;
+ }
+
+ BUG_ON(priv->tx_skbs[dirty_idx] != NULL);
+ priv->tx_skbs[dirty_idx] = skb;
+
+ /* Update the buffer descriptor with the packet information */
+ cbd_write(&bdp->len, skb->len);
+ cbd_write(&bdp->addr, skb_shinfo(skb)->dma_maps[0]);
+ cbd_write(&bdp->sc, BD_MEM_DIRTY);
+
+ if (dirty_idx == PH_RING_SIZE - 1)
+ bdp = priv->tx_base;
+ else
+ bdp++;
+
+ priv->cur_tx = bdp;
+ priv->tx_free--;
+ dev->trans_start = jiffies;
+
+ if (priv->tx_free == 0)
+ netif_stop_queue(dev);
+
+ if (priv->net_send_rx_packet_dbell)
+ wqt_raise_doorbell(priv, NET_RX_PACKET_DBELL);
+
+ spin_unlock_bh(&priv->net_lock);
+ return NETDEV_TX_OK;
+
+out_unlock:
+ spin_unlock_bh(&priv->net_lock);
+ return NETDEV_TX_BUSY;
+}
+
+static void wqt_tx_timeout(struct net_device *dev)
+{
+ struct wqt_dev *priv = netdev_priv(dev);
+
+ dev->stats.tx_errors++;
+ wqt_raise_doorbell(priv, NET_RX_PACKET_DBELL);
+}
+
+static void wqt_tx_complete(unsigned long data)
+{
+ struct net_device *dev = (struct net_device *)data;
+ struct wqt_dev *priv = netdev_priv(dev);
+ struct circ_buf_desc __iomem *bdp;
+ struct sk_buff *skb;
+ int do_wake, dirty_idx;
+
+ spin_lock_bh(&priv->net_lock);
+
+ bdp = priv->dirty_tx;
+ do_wake = 0;
+
+ while (cbd_read(&bdp->sc) == BD_MEM_FREE) {
+ dirty_idx = bdp - priv->tx_base;
+
+ skb = priv->tx_skbs[dirty_idx];
+ BUG_ON(skb == NULL);
+
+ dev->stats.tx_bytes += skb->len;
+ dev->stats.tx_packets++;
+
+ /* Unmap and free the transmitted skb */
+ skb_dma_unmap(priv->dev, skb, DMA_TO_DEVICE);
+ dev_kfree_skb_irq(skb);
+ priv->tx_skbs[dirty_idx] = NULL;
+
+ /* Invalidate the buffer descriptor */
+ cbd_write(&bdp->len, 0);
+ cbd_write(&bdp->addr, 0x0);
+ cbd_write(&bdp->sc, BD_MEM_READY);
+
+ /* Update the bdp */
+ if (dirty_idx == PH_RING_SIZE - 1)
+ bdp = priv->tx_base;
+ else
+ bdp++;
+
+ if (!priv->tx_free++)
+ do_wake = 1;
+ }
+
+ priv->dirty_tx = bdp;
+
+ spin_unlock_bh(&priv->net_lock);
+
+ if (do_wake)
+ netif_wake_queue(dev);
+}
+
+static int wqt_rx_napi(struct napi_struct *napi, int budget)
+{
+ struct wqt_dev *priv = container_of(napi, struct wqt_dev, napi);
+ struct net_device *dev = priv->ndev;
+ int dirty_idx;
+ int received = 0;
+ u32 sc, len;
+ struct sk_buff *skb, *skbn;
+ struct circ_buf_desc __iomem *bdp;
+
+ bdp = priv->cur_rx;
+
+ while (received < budget) {
+ dirty_idx = bdp - priv->rx_base;
+
+ sc = cbd_read(&bdp->sc);
+ len = cbd_read(&bdp->len);
+
+ /* Check if we are out of buffers to process */
+ if (sc != BD_MEM_DIRTY)
+ break;
+
+ skb = priv->rx_skbs[dirty_idx];
+ BUG_ON(skb == NULL);
+
+ /* Allocate a new RX skb to replace the dirty one */
+ skbn = wqt_alloc_rx_skb(priv);
+ if (unlikely(skbn == NULL)) {
+ skbn = skb;
+ dev->stats.rx_dropped++;
+ goto out_noalloc;
+ }
+
+ /* Unmap, trim, and push the skb into the network stack */
+ skb_dma_unmap(priv->dev, skb, DMA_FROM_DEVICE);
+ skb_trim(skb, len);
+ skb->protocol = eth_type_trans(skb, dev);
+ skb->ip_summed = CHECKSUM_UNNECESSARY;
+ dev->stats.rx_bytes += len;
+ dev->stats.rx_packets++;
+
+ netif_receive_skb(skb);
+ received++;
+
+out_noalloc:
+ /* Write the new skb into the buffer descriptor */
+ cbd_write(&bdp->len, PH_MAX_FRSIZE);
+ cbd_write(&bdp->addr, skb_shinfo(skbn)->dma_maps[0]);
+ cbd_write(&bdp->sc, BD_MEM_FREE);
+
+ priv->rx_skbs[dirty_idx] = skbn;
+
+ /* Update the bdp */
+ if (dirty_idx == PH_RING_SIZE - 1)
+ bdp = priv->rx_base;
+ else
+ bdp++;
+ }
+
+ priv->cur_rx = bdp;
+
+ /* We have processed all packets that the adapter had, but it
+ * was less than our budget, stop polling */
+ if (received < budget) {
+ netif_rx_complete(napi);
+ wqt_raise_doorbell(priv, NET_ENABLE_RX_PACKET_DBELL);
+ }
+
+ wqt_raise_doorbell(priv, NET_TX_COMPLETE_DBELL);
+
+ return received;
+}
+
+static const struct net_device_ops wqt_net_ops = {
+ .ndo_open = wqt_open,
+ .ndo_stop = wqt_stop,
+ .ndo_change_mtu = wqt_change_mtu,
+ .ndo_start_xmit = wqt_hard_start_xmit,
+ .ndo_tx_timeout = wqt_tx_timeout,
+};
+
+/*----------------------------------------------------------------------------*/
+/* UART Device Operations */
+/*----------------------------------------------------------------------------*/
+
+static unsigned int wqtuart_tx_empty(struct uart_port *port)
+{
+ return TIOCSER_TEMT;
+}
+
+static void wqtuart_set_mctrl(struct uart_port *port, unsigned int mctrl)
+{
+}
+
+static unsigned int wqtuart_get_mctrl(struct uart_port *port)
+{
+ return TIOCM_CAR | TIOCM_DSR | TIOCM_CTS;
+}
+
+static void wqtuart_stop_tx(struct uart_port *port)
+{
+}
+
+static void wqtuart_start_tx(struct uart_port *port)
+{
+ struct wqt_dev *priv = container_of(port, struct wqt_dev, port);
+
+ queue_work(priv->wq, &priv->uart_tx_work);
+}
+
+static void wqtuart_stop_rx(struct uart_port *port)
+{
+ struct wqt_dev *priv = container_of(port, struct wqt_dev, port);
+
+ do_disable_uart_handlers(priv);
+}
+
+static void wqtuart_enable_ms(struct uart_port *port)
+{
+}
+
+static void wqtuart_break_ctl(struct uart_port *port, int break_state)
+{
+}
+
+static int wqtuart_startup(struct uart_port *port)
+{
+ struct wqt_dev *priv = container_of(port, struct wqt_dev, port);
+ int ret;
+
+ ret = wqt_request_irq(priv);
+ if (ret)
+ return ret;
+
+ do_enable_uart_handlers(priv);
+
+ /* Mark the transmitter and receiver ready */
+ priv->uart_tx_ready = true;
+
+ /* Let the other side know that we are ready to receive chars now */
+ wqt_raise_doorbell(priv, UART_TX_EMPTY_DBELL);
+ priv->uart_open = true;
+ return 0;
+}
+
+static void wqtuart_shutdown(struct uart_port *port)
+{
+ struct wqt_dev *priv = container_of(port, struct wqt_dev, port);
+
+ wqt_free_irq(priv);
+
+ /* Make sure the uart_tx_work_fn() exits cleanly */
+ priv->uart_open = false;
+ wake_up(&priv->uart_tx_wait);
+}
+
+static void wqtuart_set_termios(struct uart_port *port,
+ struct ktermios *termios,
+ struct ktermios *old)
+{
+}
+
+static const char *wqtuart_type(struct uart_port *port)
+{
+ return "WQTUART";
+}
+
+static int wqtuart_request_port(struct uart_port *port)
+{
+ return 0;
+}
+
+static void wqtuart_config_port(struct uart_port *port, int flags)
+{
+}
+
+static void wqtuart_release_port(struct uart_port *port)
+{
+}
+
+static int wqtuart_verify_port(struct uart_port *port,
+ struct serial_struct *ser)
+{
+ return 0;
+}
+
+static struct uart_ops wqtuart_ops = {
+ .tx_empty = wqtuart_tx_empty,
+ .set_mctrl = wqtuart_set_mctrl,
+ .get_mctrl = wqtuart_get_mctrl,
+ .stop_tx = wqtuart_stop_tx,
+ .start_tx = wqtuart_start_tx,
+ .stop_rx = wqtuart_stop_rx,
+ .enable_ms = wqtuart_enable_ms,
+ .break_ctl = wqtuart_break_ctl,
+ .startup = wqtuart_startup,
+ .shutdown = wqtuart_shutdown,
+ .set_termios = wqtuart_set_termios,
+ .type = wqtuart_type,
+ .release_port = wqtuart_release_port,
+ .request_port = wqtuart_request_port,
+ .config_port = wqtuart_config_port,
+ .verify_port = wqtuart_verify_port,
+};
+
+static struct uart_driver wqtuart_driver = {
+ .owner = THIS_MODULE,
+ .driver_name = driver_name,
+ .dev_name = "ttyPCI",
+ .major = 240,
+ .minor = 0,
+ .nr = 32, /* 32 ports maximum */
+};
+
+/*----------------------------------------------------------------------------*/
+/* PCI Subsystem */
+/*----------------------------------------------------------------------------*/
+
+static int wqt_probe(struct pci_dev *dev, const struct pci_device_id *id)
+{
+ struct net_device *ndev;
+ struct wqt_dev *priv;
+ int ret, i;
+
+ ndev = alloc_etherdev(sizeof(*priv));
+ if (!ndev) {
+ ret = -ENOMEM;
+ goto out_alloc_ndev;
+ }
+
+ pci_set_drvdata(dev, ndev);
+ priv = netdev_priv(ndev);
+ priv->pdev = dev;
+ priv->dev = &dev->dev;
+ priv->ndev = ndev;
+
+ mutex_init(&priv->irq_mutex);
+ spin_lock_init(&priv->irq_lock);
+
+ /* Hardware Initialization */
+ ret = pci_enable_device(dev);
+ if (ret)
+ goto out_pci_enable_dev;
+
+ pci_set_master(dev);
+
+ ret = pci_request_regions(dev, driver_name);
+ if (ret)
+ goto out_pci_request_regions;
+
+ priv->immr = pci_ioremap_bar(dev, 0);
+ if (!priv->immr) {
+ ret = -ENOMEM;
+ goto out_ioremap_immr;
+ }
+
+ priv->netregs = pci_ioremap_bar(dev, 1);
+ if (!priv->netregs) {
+ ret = -ENOMEM;
+ goto out_ioremap_netregs;
+ }
+
+ priv->rx_base = priv->netregs + PCINET_RXBD_BASE;
+ priv->tx_base = priv->netregs + PCINET_TXBD_BASE;
+
+ ret = dma_set_mask(&dev->dev, 0xcfffffff);
+ if (ret) {
+ dev_err(&dev->dev, "Unable to set DMA mask\n");
+ ret = -ENODEV;
+ goto out_set_dma_mask;
+ }
+
+ /* Initialize private data */
+ priv->wq = create_singlethread_workqueue(driver_name);
+ if (!priv->wq) {
+ ret = -ENOMEM;
+ goto out_create_workqueue;
+ }
+
+ INIT_WORK(&priv->uart_tx_work, uart_tx_work_fn);
+ init_waitqueue_head(&priv->uart_tx_wait);
+ priv->uart_tx_ready = true;
+
+ tasklet_init(&priv->tx_complete_tasklet, wqt_tx_complete,
+ (unsigned long)ndev);
+ tasklet_disable(&priv->tx_complete_tasklet);
+ spin_lock_init(&priv->net_lock);
+
+ mutex_init(&priv->net_mutex);
+ priv->net_state = NET_STATE_STOPPED;
+ INIT_WORK(&priv->net_start_work, wqtnet_start_work_fn);
+ INIT_WORK(&priv->net_stop_work, wqtnet_stop_work_fn);
+ init_completion(&priv->net_start_completion);
+ init_completion(&priv->net_stop_completion);
+
+ /* Disable all interrupts */
+ IMMR_W32(OMIMR_OFFSET, 0x8 | 0x2 | 0x1);
+
+ /* Network Device */
+ random_ether_addr(ndev->dev_addr);
+
+ ndev->netdev_ops = &wqt_net_ops;
+ ndev->watchdog_timeo = HZ / 4;
+ ndev->flags &= ~IFF_MULTICAST; /* No multicast support */
+ ndev->mtu = PH_MAX_MTU;
+ netif_napi_add(ndev, &priv->napi, wqt_rx_napi, PH_RING_SIZE);
+
+ ret = register_netdev(ndev);
+ if (ret)
+ goto out_register_netdev;
+
+ /* UART Device */
+ priv->port.ops = &wqtuart_ops;
+ priv->port.type = PORT_16550A;
+ priv->port.dev = &dev->dev;
+ priv->port.line = 0;
+ spin_lock_init(&priv->port.lock);
+
+ /* Try to find a free uart port */
+ for (i = 0; i < wqtuart_driver.nr; i++) {
+ priv->port.line = i;
+ ret = uart_add_one_port(&wqtuart_driver, &priv->port);
+ if (ret == 0)
+ break;
+ }
+
+ if (ret)
+ goto out_add_uart_port;
+
+ dev_info(priv->dev, "using ethernet device %s\n", ndev->name);
+ dev_info(priv->dev, "using serial device %s%d\n",
+ wqtuart_driver.dev_name, priv->port.line);
+ return 0;
+
+out_add_uart_port:
+ unregister_netdev(ndev);
+out_register_netdev:
+ destroy_workqueue(priv->wq);
+out_create_workqueue:
+out_set_dma_mask:
+ iounmap(priv->netregs);
+out_ioremap_netregs:
+ iounmap(priv->immr);
+out_ioremap_immr:
+ pci_release_regions(dev);
+out_pci_request_regions:
+ pci_disable_device(dev);
+out_pci_enable_dev:
+ free_netdev(ndev);
+out_alloc_ndev:
+ return ret;
+}
+
+static void wqt_remove(struct pci_dev *dev)
+{
+ struct net_device *ndev = pci_get_drvdata(dev);
+ struct wqt_dev *priv = netdev_priv(ndev);
+
+ uart_remove_one_port(&wqtuart_driver, &priv->port);
+ unregister_netdev(priv->ndev);
+
+ flush_workqueue(priv->wq);
+ destroy_workqueue(priv->wq);
+
+ /* Disable all interrupts */
+ IMMR_W32(OMIMR_OFFSET, 0x8 | 0x2 | 0x1);
+
+ iounmap(priv->netregs);
+ iounmap(priv->immr);
+ pci_release_regions(dev);
+ pci_disable_device(dev);
+
+ free_netdev(ndev);
+}
+
+#define PCI_DEVID_FSL_MPC8349EMDS 0x0080
+
+/* The list of devices that this module will support */
+static struct pci_device_id wqt_ids[] = {
+ { PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, PCI_DEVID_FSL_MPC8349EMDS), },
+ { 0, }
+};
+MODULE_DEVICE_TABLE(pci, wqt_ids);
+
+static struct pci_driver wqt_pci_driver = {
+ .name = (char *)driver_name,
+ .id_table = wqt_ids,
+ .probe = wqt_probe,
+ .remove = wqt_remove,
+};
+
+/*----------------------------------------------------------------------------*/
+/* Module Init / Exit */
+/*----------------------------------------------------------------------------*/
+
+static int __init wqt_init(void)
+{
+ int ret;
+
+ ret = uart_register_driver(&wqtuart_driver);
+ if (ret)
+ goto out_uart_register_driver;
+
+ ret = pci_register_driver(&wqt_pci_driver);
+ if (ret)
+ goto out_pci_register_driver;
+
+ return 0;
+
+out_pci_register_driver:
+ uart_unregister_driver(&wqtuart_driver);
+out_uart_register_driver:
+ return ret;
+}
+
+static void __exit wqt_exit(void)
+{
+ pci_unregister_driver(&wqt_pci_driver);
+ uart_unregister_driver(&wqtuart_driver);
+}
+
+MODULE_AUTHOR("Ira W. Snyder <[email protected]>");
+MODULE_DESCRIPTION("PCINet/PCISerial Driver for MPC8349EMDS (Host side)");
+MODULE_LICENSE("GPL");
+
+module_init(wqt_init);
+module_exit(wqt_exit);
diff --git a/drivers/net/pcinet_hw.h b/drivers/net/pcinet_hw.h
new file mode 100644
index 0000000..499ba61
--- /dev/null
+++ b/drivers/net/pcinet_hw.h
@@ -0,0 +1,77 @@
+/*
+ * Register offsets for the MPC8349EMDS Message Unit from the IMMR base address
+ *
+ * Copyright (c) 2008 Ira W. Snyder <[email protected]>
+ *
+ * This file is licensed under the terms of the GNU General Public License
+ * version 2. This program is licensed "as is" without any warranty of any
+ * kind, whether express or implied.
+ */
+
+#ifndef PCINET_HW_H
+#define PCINET_HW_H
+
+/* mpc8349emds message unit register offsets */
+#define OMISR_OFFSET 0x8030
+#define OMIMR_OFFSET 0x8034
+#define IMR0_OFFSET 0x8050
+#define IMR1_OFFSET 0x8054
+#define OMR0_OFFSET 0x8058
+#define OMR1_OFFSET 0x805C
+#define ODR_OFFSET 0x8060
+#define IDR_OFFSET 0x8068
+#define IMISR_OFFSET 0x8080
+#define IMIMR_OFFSET 0x8084
+
+
+/* mpc8349emds pci and local access window register offsets */
+#define LAWAR0_OFFSET 0x0064
+#define LAWAR0_ENABLE (1<<31)
+
+#define POCMR0_OFFSET 0x8410
+#define POCMR0_ENABLE (1<<31)
+
+#define POTAR0_OFFSET 0x8400
+
+#define LAWAR1_OFFSET 0x006c
+#define LAWAR1_ENABLE (1<<31)
+
+#define POCMR1_OFFSET 0x8428
+#define POCMR1_ENABLE (1<<31)
+
+#define POTAR1_OFFSET 0x8418
+
+
+/* mpc8349emds dma controller register offsets */
+#define DMAMR0_OFFSET 0x8100
+#define DMASR0_OFFSET 0x8104
+#define DMASAR0_OFFSET 0x8110
+#define DMADAR0_OFFSET 0x8118
+#define DMABCR0_OFFSET 0x8120
+
+#define DMA_CHANNEL_BUSY (1<<2)
+
+#define DMA_DIRECT_MODE_SNOOP (1<<20)
+#define DMA_CHANNEL_MODE_DIRECT (1<<2)
+#define DMA_CHANNEL_START (1<<0)
+
+
+/* mpc8349emds pci and local access window register offsets */
+#define LAWAR0_OFFSET 0x0064
+#define LAWAR0_ENABLE (1<<31)
+
+#define POCMR0_OFFSET 0x8410
+#define POCMR0_ENABLE (1<<31)
+
+#define POTAR0_OFFSET 0x8400
+
+
+/* mpc8349emds pci and inbound window register offsets */
+#define PITAR0_OFFSET 0x8568
+#define PIWAR0_OFFSET 0x8578
+
+#define PIWAR0_ENABLED (1<<31)
+#define PIWAR0_PREFETCH (1<<29)
+#define PIWAR0_IWS_4K 0xb
+
+#endif /* PCINET_HW_H */
--
1.5.4.3

2009-01-08 19:16:31

by David Miller

Subject: Re: [PATCH RFC v5] net: add PCINet driver

From: Ira Snyder <[email protected]>
Date: Wed, 7 Jan 2009 11:50:52 -0800

> This adds support to Linux for a virtual ethernet interface which uses the
> PCI bus as its transport mechanism. It creates a simple, familiar, and fast
> method of communication for two devices connected by a PCI interface.

Well, it looks like much more than that to me.

What is this UART thing in here for?

I can only assume it's meant to be used as a console port between the
x86 host and the powerpc nodes.

You haven't even mentioned this UART aspect even indirectly in the
commit message.

This just looks like yet another set of virtualization drivers
to me. You could have just have easily built this using your
own PCI backplane framework, and using the virtio stuff on top.

And the virtio stuff has all kinds of snazzy optimizations that
will likely improve your throughput, it has console drivers that
distributions already probe for and attach appropriately, etc.

In short I really don't like this conceptually, it can be done
so much better using facilities we already have that are
heavily optimized and userland understands already.

2009-01-08 19:27:39

by Ira W. Snyder

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Thu, Jan 08, 2009 at 11:16:10AM -0800, David Miller wrote:
> From: Ira Snyder <[email protected]>
> Date: Wed, 7 Jan 2009 11:50:52 -0800
>
> > This adds support to Linux for a virtual ethernet interface which uses the
> > PCI bus as its transport mechanism. It creates a simple, familiar, and fast
> > method of communication for two devices connected by a PCI interface.
>
> Well, it looks like much more than that to me.
>
> What is this UART thing in here for?
>
> I can only assume it's meant to be used as a console port between the
> x86 host and the powerpc nodes.
>

Exactly right. I needed it to tell the U-Boot bootloader to tftp the
kernel and boot the board. These boards don't keep their PCI settings
(assigned by the BIOS at boot) over a soft or hard reset.

I couldn't think of a better way to get them started.

> You haven't even mentioned this UART aspect even indirectly in the
> commit message.
>

Sorry, I'll add something about it.

> This just looks like yet another set of virtualization drivers
> to me. You could have just have easily built this using your
> own PCI backplane framework, and using the virtio stuff on top.
>
> And the virtio stuff has all kinds of snazzy optimizations that
> will likely improve your throughput, it has console drivers that
> distributions already probe for and attach appropriately, etc.
>
> In short I really don't like this conceptually, it can be done
> so much better using facilities we already have that are
> heavily optimized and userland understands already.

I've had a really hard time understanding the virtio code. I haven't
been able to find much in the way of documentation for it. Can you point
me to some code that does anything vaguely similar to what you are
suggesting?

Arnd Bergmann said that there is a similar driver (for different
hardware) that uses virtio, but it is very slow, because it doesn't use
the DMA controller to transfer data. I need at very minimum 40MB/sec of
client -> host data transfer.

Thanks for the comments,
Ira

> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2009-01-08 21:51:46

by Ira W. Snyder

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Thu, Jan 08, 2009 at 11:27:16AM -0800, Ira Snyder wrote:
> On Thu, Jan 08, 2009 at 11:16:10AM -0800, David Miller wrote:
> > From: Ira Snyder <[email protected]>
> > Date: Wed, 7 Jan 2009 11:50:52 -0800
> >
> > > This adds support to Linux for a virtual ethernet interface which uses the
> > > PCI bus as its transport mechanism. It creates a simple, familiar, and fast
> > > method of communication for two devices connected by a PCI interface.
> >
> > Well, it looks like much more than that to me.
> >
> > What is this UART thing in here for?
> >
> > I can only assume it's meant to be used as a console port between the
> > x86 host and the powerpc nodes.
> >
>
> Exactly right. I needed it to tell the U-Boot bootloader to tftp the
> kernel and boot the board. These boards don't keep their PCI settings
> (assigned by the BIOS at boot) over a soft or hard reset.
>
> I couldn't think of a better way to get them started.
>
> > You haven't even mentioned this UART aspect even indirectly in the
> > commit message.
> >
>
> Sorry, I'll add something about it.
>
> > This just looks like yet another set of virtualization drivers
> > to me. You could have just have easily built this using your
> > own PCI backplane framework, and using the virtio stuff on top.
> >
> > And the virtio stuff has all kinds of snazzy optimizations that
> > will likely improve your throughput, it has console drivers that
> > distributions already probe for and attach appropriately, etc.
> >
> > In short I really don't like this conceptually, it can be done
> > so much better using facilities we already have that are
> > heavily optimized and userland understands already.
>
> I've had a really hard time understanding the virtio code. I haven't
> been able to find much in the way of documentation for it. Can you point
> me to some code that does anything vaguely similar to what you are
> suggesting?
>
> Arnd Bergmann said that there is a similar driver (for different
> hardware) that uses virtio, but it is very slow, because it doesn't use
> the DMA controller to transfer data. I need at very minimum 40MB/sec of
> client -> host data transfer.
>

Rusty, since you wrote the virtio code, can you point me at the things I
would need to implement to use virtio over the PCI bus.

The guests (PowerPC computers running Linux) are PCI cards in the host
system (an Intel Pentium3-M system). The guest computers can access all
of the host's memory. The guests provide a 1MB (movable) window into
their memory.

The PowerPC computers also have a DMA controller, which I've used to get
better throughput from my driver. I have a way to create interrupts to
both the host and guest systems.

I've read your paper titled: "virtio: Towards a De-Facto Standard For
Virtual I/O Devices"

If I read that correctly, then I should implement all of the functions
in struct virtqueue_ops appropriately, and the existing virtio drivers
should just work. The only concern I have there is how to make guest ->
host transfer use the DMA controller. I've done all programming of it
from the guest kernel, using the Linux DMAEngine API.

Are there any other implementations other than virtio_ring?

I appreciate any input you can give. If I should be CCing someone else,
just let me know and I'll drop you from the CC list.

Thanks,
Ira

2009-01-10 23:33:29

by Benjamin Herrenschmidt

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Thu, 2009-01-08 at 13:51 -0800, Ira Snyder wrote:
> The guests (PowerPC computers running Linux) are PCI cards in the host
> system (an Intel Pentium3-M system). The guest computers can access all
> of the host's memory. The guests provide a 1MB (movable) window into
> their memory.
>
> The PowerPC computers also have a DMA controller, which I've used to get
> better throughput from my driver. I have a way to create interrupts to
> both the host and guest systems.

That looks -very- similar to the PCI driver for CAB and Cell triblades
that was, I think, submitted a while ago. Arnd what's the status with
that driver ?

Cheers,
Ben.

2009-01-12 17:56:52

by Arnd Bergmann

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Sunday 11 January 2009, Benjamin Herrenschmidt wrote:
> On Thu, 2009-01-08 at 13:51 -0800, Ira Snyder wrote:
> > The guests (PowerPC computers running Linux) are PCI cards in the host
> > system (an Intel Pentium3-M system). The guest computers can access all
> > of the host's memory. The guests provide a 1MB (movable) window into
> > their memory.
> >
> > The PowerPC computers also have a DMA controller, which I've used to get
> > better throughput from my driver. I have a way to create interrupts to
> > both the host and guest systems.
>
> That looks -very- similar to the PCI driver for CAB and Cell triblades
> that was, I think, submitted a while ago. Arnd what's the status with
> that driver ?

Complicated ;-)

There were two device drivers for the Axon PCIe endpoint (from Mercury
and IBM), but both were abandoned due to being too complex to get into
a mergeable stage, and all the original developers are doing other work
now.

I'm maintaining the IBM driver now, but have no plans to submit that
for inclusion. I hope that I can soon find more time to work on a
replacement driver that will provide the right abstraction through
virtio for kernel drivers, plus a ibverbs user interface for direct
application programming. It should also have pluggable backends to
support not only Cell hardware but anything with a similar hardware
implementation.

I did make a presentation about this at the Plumbers Conference, see
http://userweb.kernel.org/%7Earnd/papers/plumbers08/plumbers-slides.pdf.
Since then, I have come to a much clearer idea of what needs to be
done, but not more actual code.

Arnd <><

2009-01-13 02:33:22

by Rusty Russell

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Friday 09 January 2009 08:21:27 Ira Snyder wrote:
> Rusty, since you wrote the virtio code, can you point me at the things I
> would need to implement to use virtio over the PCI bus.
>
> The guests (PowerPC computers running Linux) are PCI cards in the host
> system (an Intel Pentium3-M system). The guest computers can access all
> of the host's memory. The guests provide a 1MB (movable) window into
> their memory.
>
> The PowerPC computers also have a DMA controller, which I've used to get
> better throughput from my driver. I have a way to create interrupts to
> both the host and guest systems.
>
> I've read your paper titled: "virtio: Towards a De-Facto Standard For
> Virtual I/O Devices"
>
> If I read that correctly, then I should implement all of the functions
> in struct virtqueue_ops appropriately, and the existing virtio drivers
> should just work. The only concern I have there is how to make guest ->
> host transfer use the DMA controller. I've done all programming of it
> from the guest kernel, using the Linux DMAEngine API.

Hi Ira,

Interesting system: the guest being able to access the host's memory but not (fully) vice-versa makes this a little different from the current implementations where that was assumed. virtio assumes that the guest will publish buffers and someone else (ie. the host) will access them.

> Are there any other implementations other than virtio_ring?

There were some earlier implementations, but they were trivial. And not particularly helpful in your case.

In summary, yes, it should work quite nicely, but rather than publishing the buffers to the host, the host will need to tell the guest where it wants them transferred (possibly using that 1M window, and a notification mechanism). That will be the complex part I think.

Cheers,
Rusty.

2009-01-13 03:34:35

by Ira W. Snyder

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Tue, Jan 13, 2009 at 01:02:52PM +1030, Rusty Russell wrote:
> On Friday 09 January 2009 08:21:27 Ira Snyder wrote:
> > Rusty, since you wrote the virtio code, can you point me at the things I
> > would need to implement to use virtio over the PCI bus.
> >
> > The guests (PowerPC computers running Linux) are PCI cards in the host
> > system (an Intel Pentium3-M system). The guest computers can access all
> > of the host's memory. The guests provide a 1MB (movable) window into
> > their memory.
> >
> > The PowerPC computers also have a DMA controller, which I've used to get
> > better throughput from my driver. I have a way to create interrupts to
> > both the host and guest systems.
> >
> > I've read your paper titled: "virtio: Towards a De-Facto Standard For
> > Virtual I/O Devices"
> >
> > If I read that correctly, then I should implement all of the functions
> > in struct virtqueue_ops appropriately, and the existing virtio drivers
> > should just work. The only concern I have there is how to make guest ->
> > host transfer use the DMA controller. I've done all programming of it
> > from the guest kernel, using the Linux DMAEngine API.
>
> Hi Ira,
>
> Interesting system: the guest being able to access the host's memory but not (fully) vice-versa makes this a little different from the current implementations where that was assumed. virtio assumes that the guest will publish buffers and someone else (ie. the host) will access them.
>

The guest system /could/ publish all of its RAM, but with 256MB per
board, 19 boards per cPCI crate, that's way too much for a 32-bit PC to
map into it's memory space. That's the real reason I use the 1MB
windows. I could make them bigger (16MB would be fine, I think), but I
doubt it would make much of a difference to the implementation.

> > Are there any other implementations other than virtio_ring?
>
> There were some earlier implementations, but they were trivial. And not particularly helpful in your case.
>
> In summary, yes, it should work quite nicely, but rather than publishing the buffers to the host, the host will need to tell the guest where it wants them transferred (possibly using that 1M window, and a notification mechanism). That will be the complex part I think.
>

I've got a system like that right now in the driver that started this
thread. I tried to mimic the buffer descriptor rings in a network card.
The virtio system doesn't seem to be too different. I can probably work
something out.

What's keeping me from getting started is what to do with the
virtqueue_ops structure once I've got the functions to fill it in. There
isn't a register_virtqueue_ops() function. I'm not at all sure when to
use a struct virtio_driver or a struct virtio_device. I hunch I need one
of these.

If I've understood this whole thing correctly, then I should be able to
register a virtio_ops on both sides, and then insmod virtio_net and
virtio_console, and have them "just work".

I guess what I'm looking for is something trivial, something that'll
insmod and printk() whenever one of the virtqueue_ops pointers is
called. Something like a template. That's how I really learned when all
of the uart functions are called, for example.

Thanks,
Ira

2009-01-13 16:33:37

by Arnd Bergmann

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Tuesday 13 January 2009, Ira Snyder wrote:
> On Tue, Jan 13, 2009 at 01:02:52PM +1030, Rusty Russell wrote:
> >
> > Interesting system: the guest being able to access the
> > host's memory but not (fully) vice-versa makes this a
> > little different from the current implementations where
> > that was assumed. virtio assumes that the guest will
> > publish buffers and someone else (ie. the host) will access them.
>
> The guest system /could/ publish all of its RAM, but with 256MB per
> board, 19 boards per cPCI crate, that's way too much for a 32-bit PC to
> map into it's memory space. That's the real reason I use the 1MB
> windows. I could make them bigger (16MB would be fine, I think), but I
> doubt it would make much of a difference to the implementation.

The way we do it in the existing driver for cell, both sides export
just a little part of their memory to the other side, and they
also both get access to one channel of the DMA engine, which is
enough to transfer larger data sections, as the DMA engine has
access to all the memory on both sides.

Arnd <><

2009-01-13 16:40:37

by Ira W. Snyder

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Tue, Jan 13, 2009 at 05:33:03PM +0100, Arnd Bergmann wrote:
> On Tuesday 13 January 2009, Ira Snyder wrote:
> > On Tue, Jan 13, 2009 at 01:02:52PM +1030, Rusty Russell wrote:
> > >
> > > Interesting system: the guest being able to access the
> > > host's memory but not (fully) vice-versa makes this a
> > > little different from the current implementations where
> > > that was assumed. virtio assumes that the guest will
> > > publish buffers and someone else (ie. the host) will access them.
> >
> > The guest system /could/ publish all of its RAM, but with 256MB per
> > board, 19 boards per cPCI crate, that's way too much for a 32-bit PC to
> > map into it's memory space. That's the real reason I use the 1MB
> > windows. I could make them bigger (16MB would be fine, I think), but I
> > doubt it would make much of a difference to the implementation.
>
> The way we do it in the existing driver for cell, both sides export
> just a little part of their memory to the other side, and they
> also both get access to one channel of the DMA engine, which is
> enough to transfer larger data sections, as the DMA engine has
> access to all the memory on both sides.

So do you program one channel of the DMA engine from the host side and
another channel from the guest side?

I tried to avoid having the host program the DMA controller at all.
Using the DMAEngine API on the guest did better than I could achieve by
programming the registers manually. I didn't use chaining or any of the
fancier features in my tests, though.

Ira

2009-01-13 17:43:33

by Arnd Bergmann

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Tuesday 13 January 2009, Ira Snyder wrote:
> So do you program one channel of the DMA engine from the host side and
> another channel from the guest side?

Yes.

> I tried to avoid having the host program the DMA controller at all.
> Using the DMAEngine API on the guest did better than I could achieve by
> programming the registers manually. I didn't use chaining or any of the
> fancier features in my tests, though.

Our driver unfortunately does not use the DMA API, but it clearly
should. What this means in your case is that you would need to port
the freescale DMA engine driver to the host side, to operate on the
PCI device. Not sure about how specifically this would work on fsl
hardware, but it seems generally doable.

Arnd <><

2009-01-15 00:13:15

by Ira W. Snyder

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Tue, Jan 13, 2009 at 06:42:53PM +0100, Arnd Bergmann wrote:
> On Tuesday 13 January 2009, Ira Snyder wrote:
> > So do you program one channel of the DMA engine from the host side and
> > another channel from the guest side?
>
> Yes.
>
> > I tried to avoid having the host program the DMA controller at all.
> > Using the DMAEngine API on the guest did better than I could achieve by
> > programming the registers manually. I didn't use chaining or any of the
> > fancier features in my tests, though.
>
> Our driver unfortunately does not use the DMA API, but it clearly
> should. What this means in your case is that you would need to port
> the freescale DMA engine driver to the host side, to operate on the
> PCI device. Not sure about how specifically this would work on fsl
> hardware, but it seems generally doable.
>

The only problem with that is that you cannot route interrupts from the
DMA controller over PCI with the PowerPC core running. Which makes it
mostly useless for this case.

It /could/ work with polling mmio reads of the DMA controller register,
but as you said, mmio reads are slow, and that would be best avoided. I
don't think porting the driver is the right way to go.

If I haven't said it before, let me say it now: the DMA controller can
access all of the host's memory as well as all of the guest's memory.



I have been studying the virtio and virtio_ring systems, and I now have
a good idea about how to implement virtio-over-pci. I'm having trouble
coming up with a good scheme to use the DMA controller for all the data
transfers, though.

It is fairly easy to use the DMA controller in the virtio_net case,
since the virtio_net driver allocates all of the RX buffers ahead of
time. The guest can DMA directly from host TX into guest RX, then mark
the host TX buffers consumed. The same is true in reverse: the guest can
DMA directly from it's TX into the host RX, then mark the host RX
buffers consumed.

I'm not sure how to handle the bidirectional virtqueue case, though.
Such as the upcoming changes to virtio_net or virtio_blk. It seems like
you'd need to copy pages in this case, since you want to use the DMA to
transfer them.

I have come up with a nice scheme to maintain the buffer descriptor
tables in both host and guest memory without doing anything except mmio
writes. The scheme is almost identical to virtio_ring, with the
exception that both machines need a copy of vring->desc's flags and next
fields, to avoid the reads in virtio_rings detach_buf().

Maybe I'm overoptimistic in thinking that two instances of virtio_net
could talk to each other if the RX and TX queues were hooked up
accordingly in the find_vq() function. Has anyone actually tried this?



To anyone that is trying to learn virtio, I dummied up this mock driver
that registers a virtio_device for every MPC8349EMDS board in the
system. The virtqueues don't actually do anything, but they do print
messages so you know when they're used. It is almost certainly buggy and
horrible code. I'm only reproducing it here because it helped me get
started.

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/init.h>
#include <linux/pci.h>
#include <linux/virtio.h>
#include <linux/virtio_config.h>
#include <linux/virtio_net.h>

#include "pcinet_hw.h"

static const char driver_name[] = "vdev";

struct vpt_dev {
struct virtio_device vdev;
u8 status;

void __iomem *immr;
void __iomem *netregs;

#define NUM_VQS 4
struct virtqueue *vqs[NUM_VQS];

unsigned int in;
unsigned int out;
};

#define vdev_to_vpt(vdev) container_of(vdev, struct vpt_dev, vdev)

/*----------------------------------------------------------------------------*/
/* struct virtqueue_ops infrastructure */
/*----------------------------------------------------------------------------*/

static int vpt_vq_add_buf(struct virtqueue *vq,
struct scatterlist sg[],
unsigned int out_num,
unsigned int in_num,
void *data)
{
struct virtio_device *vdev = vq->vdev;
struct vpt_dev *priv = vdev_to_vpt(vdev);

pr_info("%s: vq=0x%p sg=0x%p, out=%u in=%d data=0x%p\n",
__func__, vq, sg, out_num, in_num, data);

priv->in += in_num;
priv->out += out_num;

if (priv->in > 10 || priv->out > 10) {
pr_info("%s: vq=0x%p buffers filled\n", __func__, vq);
return -ENOMEM;
}

return 0;
}

static void *vpt_vq_get_buf(struct virtqueue *vq, unsigned int *len)
{
pr_info("%s: vq=0x%p\n", __func__, vq);
return NULL;
}

static void vpt_vq_kick(struct virtqueue *vq)
{
pr_info("%s: vq=0x%p\n", __func__, vq);
}

static void vpt_vq_disable_cb(struct virtqueue *vq)
{
pr_info("%s: vq=0x%p\n", __func__, vq);
}

static bool vpt_vq_enable_cb(struct virtqueue *vq)
{
pr_info("%s: vq=0x%p\n", __func__, vq);
return true;
}

static struct virtqueue_ops vpt_vq_ops = {
.add_buf = vpt_vq_add_buf,
.get_buf = vpt_vq_get_buf,
.kick = vpt_vq_kick,
.disable_cb = vpt_vq_disable_cb,
.enable_cb = vpt_vq_enable_cb,
};

/*----------------------------------------------------------------------------*/
/* struct virtio_device infrastructure */
/*----------------------------------------------------------------------------*/

static void vptc_get(struct virtio_device *vdev,
unsigned offset,
void *buf,
unsigned len)
{
pr_info("%s: vdev=0x%p offset=%u buf=0x%p len=%u\n",
__func__, vdev, offset, buf, len);
}

static void vptc_set(struct virtio_device *vdev,
unsigned offset,
const void *buf,
unsigned len)
{
pr_info("%s: vdev=0x%p offset=%u buf=%p len=%u\n",
__func__, vdev, offset, buf, len);
}

static u8 vptc_get_status(struct virtio_device *vdev)
{
struct vpt_dev *priv = vdev_to_vpt(vdev);

pr_info("%s: vdev=0x%p\n", __func__, vdev);

return priv->status;
}

static void vptc_set_status(struct virtio_device *vdev, u8 status)
{
struct vpt_dev *priv = vdev_to_vpt(vdev);

pr_info("%s: vdev=0x%p status=0x%.2x\n", __func__, vdev, status);

priv->status = status;
}

static void vptc_reset(struct virtio_device *vdev)
{
struct vpt_dev *priv = vdev_to_vpt(vdev);

pr_info("%s: vdev=0x%p\n", __func__, vdev);

priv->status = 0;
}

static struct virtqueue *vptc_find_vq(struct virtio_device *vdev,
unsigned index,
void (*cb)(struct virtqueue *vq))
{
struct vpt_dev *priv = vdev_to_vpt(vdev);
struct virtqueue *vq = priv->vqs[index];

pr_info("%s: vdev=0x%p index=%u cb=0x%p\n", __func__, vdev, index, cb);

if (vq)
return vq;

vq = kzalloc(sizeof(*vq), GFP_KERNEL);
if (!vq) {
pr_info("%s: vdev=0x%p unable to allocate vq\n", __func__, vq);
return ERR_PTR(-ENOMEM);
}

vq->callback = cb;
vq->vdev = vdev;
vq->vq_ops = &vpt_vq_ops;

pr_info("%s: vdev=0x%p new vq=0x%p\n", __func__, vdev, vq);

priv->vqs[index] = vq;
return vq;
}

static void vptc_del_vq(struct virtqueue *vq)
{
struct virtio_device *vdev = vq->vdev;
struct vpt_dev *priv = vdev_to_vpt(vdev);
int i;

pr_info("%s: vq=0x%p\n", __func__, vq);

for (i = 0; i < NUM_VQS; i++) {
if (priv->vqs[i] == vq) {
kfree(vq);
priv->vqs[i] = NULL;
return;
}
}

pr_err("%s: vdev=0x%p unable to find vq=0x%p\n", __func__, vdev, vq);
}

static u32 vptc_get_features(struct virtio_device *vdev)
{
pr_info("%s: vdev=0x%p\n", __func__, vdev);
return 0;
}

static void vptc_finalize_features(struct virtio_device *vdev)
{
pr_info("%s: vdev=0x%p\n", __func__, vdev);
}

static struct virtio_config_ops vpt_config_ops = {
.get = vptc_get,
.set = vptc_set,
.get_status = vptc_get_status,
.set_status = vptc_set_status,
.reset = vptc_reset,
.find_vq = vptc_find_vq,
.del_vq = vptc_del_vq,
.get_features = vptc_get_features,
.finalize_features = vptc_finalize_features,
};

/*----------------------------------------------------------------------------*/
/* PCI Subsystem */
/*----------------------------------------------------------------------------*/

static int vpt_probe(struct pci_dev *dev, const struct pci_device_id *id)
{
struct vpt_dev *priv;
int ret;

priv = kzalloc(sizeof(*priv), GFP_KERNEL);
if (!priv) {
ret = -ENOMEM;
goto out_return;
}

pci_set_drvdata(dev, priv);

/* Hardware Initialization */
ret = pci_enable_device(dev);
if (ret)
goto out_pci_enable_dev;

pci_set_master(dev);
ret = pci_request_regions(dev, driver_name);
if (ret)
goto out_pci_request_regions;

priv->immr = pci_ioremap_bar(dev, 0);
if (!priv->immr) {
ret = -ENOMEM;
goto out_ioremap_immr;
}

priv->netregs = pci_ioremap_bar(dev, 1);
if (!priv->netregs) {
ret = -ENOMEM;
goto out_ioremap_netregs;
}

ret = dma_set_mask(&dev->dev, DMA_BIT_MASK(30));
if (ret) {
dev_err(&dev->dev, "Unable to set DMA mask\n");
ret = -ENODEV;
goto out_set_dma_mask;
}

/* Mask all of the MBOX interrupts */
iowrite32(0x1 | 0x2, priv->immr + OMIMR_OFFSET);

/* Register the virtio_device corresponding to this PCI device */
priv->vdev.dev.parent = &dev->dev;
priv->vdev.id.device = VIRTIO_ID_NET;
priv->vdev.config = &vpt_config_ops;

ret = register_virtio_device(&priv->vdev);
if (ret) {
dev_err(&dev->dev, "Unable to register virtio device\n");
goto out_register_virtio_device;
}

return 0;

out_register_virtio_device:
out_set_dma_mask:
iounmap(priv->netregs);
out_ioremap_netregs:
iounmap(priv->immr);
out_ioremap_immr:
pci_release_regions(dev);
out_pci_request_regions:
pci_disable_device(dev);
out_pci_enable_dev:
kfree(priv);
out_return:
return ret;
}

static void vpt_remove(struct pci_dev *dev)
{
struct vpt_dev *priv = pci_get_drvdata(dev);

unregister_virtio_device(&priv->vdev);
iounmap(priv->netregs);
iounmap(priv->immr);
pci_release_regions(dev);
pci_disable_device(dev);
kfree(priv);
}

#define PCI_DEVID_FSL_MPC8349EMDS 0x0080

/* The list of devices that this module will support */
static struct pci_device_id vpt_ids[] = {
{ PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, PCI_DEVID_FSL_MPC8349EMDS), },
{ 0, }
};
MODULE_DEVICE_TABLE(pci, vpt_ids);

static struct pci_driver vpt_pci_driver = {
.name = (char *)driver_name,
.id_table = vpt_ids,
.probe = vpt_probe,
.remove = vpt_remove,
};

/*----------------------------------------------------------------------------*/
/* Module Init / Exit */
/*----------------------------------------------------------------------------*/

static int __init vpt_init(void)
{
return pci_register_driver(&vpt_pci_driver);
}

static void __exit vpt_exit(void)
{
pci_unregister_driver(&vpt_pci_driver);
}

MODULE_AUTHOR("Ira W. Snyder <[email protected]>");
MODULE_DESCRIPTION("Virtio PCI Test Driver");
MODULE_LICENSE("GPL");

module_init(vpt_init);
module_exit(vpt_exit);

2009-01-15 12:58:56

by Arnd Bergmann

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Thursday 15 January 2009, Ira Snyder wrote:

> The only problem with that is that you cannot route interrupts from the
> DMA controller over PCI with the PowerPC core running. Which makes it
> mostly useless for this case.

If the host supports MSI, you can simply program the DMA controller to
write the correct message to the inbound address of the MSI interrupt
controller!

All modern host systems should have MSI, as this is required by the
PCIe specification, but it still somewhat limits the choice of your
hosts.

Arnd <><

2009-01-15 16:54:20

by Ira W. Snyder

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Thu, Jan 15, 2009 at 01:58:33PM +0100, Arnd Bergmann wrote:
> On Thursday 15 January 2009, Ira Snyder wrote:
>
> > The only problem with that is that you cannot route interrupts from the
> > DMA controller over PCI with the PowerPC core running. Which makes it
> > mostly useless for this case.
>
> If the host supports MSI, you can simply program the DMA controller to
> write the correct message to the inbound address of the MSI interrupt
> controller!
>
> All modern host systems should have MSI, as this is required by the
> PCIe specification, but it still somewhat limits the choice of your
> hosts.
>

These are PCI boards, not PCIe. The host computers are all Pentium3-M
systems. I tried enabling MSI on the Freescale boards in the driver, by
calling pci_enable_msi() during probe(), and it failed. I'm pretty sure
I can't do MSI.

I'll keep thinking about how to manage the DMA. Feel free to toss around
any ideas you have, though.

Ira

2009-01-15 17:54:19

by Arnd Bergmann

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Thursday 15 January 2009, Ira Snyder wrote:
>
> These are PCI boards, not PCIe. The host computers are all Pentium3-M
> systems. I tried enabling MSI on the Freescale boards in the driver, by
> calling pci_enable_msi() during probe(), and it failed.

That doesn't really mean anything, just that the PCI endpoint doesn't
announce its capability to do MSI in the config space, or that it
does not have an interrupt line. Since you basically implement the
device on the FSL board, you should also be able to define the interrupt
capabilities by writing to the config space.

Do you know what kind of chipset the host uses? It should be fairly
simple to find out whether or not it can do MSI.

Arnd <><

2009-01-15 18:20:54

by Ira W. Snyder

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Thu, Jan 15, 2009 at 06:53:51PM +0100, Arnd Bergmann wrote:
> On Thursday 15 January 2009, Ira Snyder wrote:
> >
> > These are PCI boards, not PCIe. The host computers are all Pentium3-M
> > systems. I tried enabling MSI on the Freescale boards in the driver, by
> > calling pci_enable_msi() during probe(), and it failed.
>
> That doesn't really mean anything, just that the PCI endpoint doesn't
> announce its capability to do MSI in the config space, or that it
> does not have an interrupt line. Since you basically implement the
> device on the FSL board, you should also be able to define the interrupt
> capabilities by writing to the config space.
>
> Do you know what kind of chipset the host uses? It should be fairly
> simple to find out whether or not it can do MSI.
>

Some sort of Broadcom chipset, I think. Full dmesg and lspci output are
appended.

The PCI bridge does mention MSI, so maybe it does support it. Would
using the DMA from the host mean that the guest system couldn't use the
DMA controller at all? All of the channels share the same interrupt line
on the guest. I need one channel on the DMA controller on the guest to
do realtime transfers from some data processing FPGAs on the board.

This stuff is never easy, is it? :)

Thanks for the help so far, I really appreciate it.
Ira


[ 0.000000] Linux version 2.6.28-07812-g5fbbf5f (iws@labslcor1) (gcc version 4.1.2 (Gentoo 4.1.2 p1.1)) #4 Thu Jan 8 15:45:08 PST 2009
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] NSC Geode by NSC
[ 0.000000] Cyrix CyrixInstead
[ 0.000000] Centaur CentaurHauls
[ 0.000000] Transmeta GenuineTMx86
[ 0.000000] Transmeta TransmetaCPU
[ 0.000000] UMC UMC UMC UMC
[ 0.000000] PAT WC disabled due to known CPU erratum.
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
[ 0.000000] BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
[ 0.000000] BIOS-e820: 00000000000ea800 - 0000000000100000 (reserved)
[ 0.000000] BIOS-e820: 0000000000100000 - 0000000020000000 (usable)
[ 0.000000] BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
[ 0.000000] DMI present.
[ 0.000000] Phoenix BIOS detected: BIOS may corrupt low RAM, working it around.
[ 0.000000] last_pfn = 0x20000 max_arch_pfn = 0x100000
[ 0.000000] Scanning 0 areas for low memory corruption
[ 0.000000] modified physical RAM map:
[ 0.000000] modified: 0000000000000000 - 0000000000010000 (reserved)
[ 0.000000] modified: 0000000000010000 - 000000000009f800 (usable)
[ 0.000000] modified: 000000000009f800 - 00000000000a0000 (reserved)
[ 0.000000] modified: 00000000000ea800 - 0000000000100000 (reserved)
[ 0.000000] modified: 0000000000100000 - 0000000020000000 (usable)
[ 0.000000] modified: 00000000fff80000 - 0000000100000000 (reserved)
[ 0.000000] kernel direct mapping tables up to 20000000 @ 10000-15000
[ 0.000000] ACPI Error (tbxfroot-0218): A valid RSDP was not found [20080926]
[ 0.000000] 512MB LOWMEM available.
[ 0.000000] mapped low ram: 0 - 20000000
[ 0.000000] low ram: 00000000 - 20000000
[ 0.000000] bootmap 00011000 - 00015000
[ 0.000000] (6 early reservations) ==> bootmem [0000000000 - 0020000000]
[ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
[ 0.000000] #1 [0000100000 - 00007bf49c] TEXT DATA BSS ==> [0000100000 - 00007bf49c]
[ 0.000000] #2 [00007c0000 - 00007c3000] INIT_PG_TABLE ==> [00007c0000 - 00007c3000]
[ 0.000000] #3 [000008f400 - 0000100000] BIOS reserved ==> [000008f400 - 0000100000]
[ 0.000000] #4 [0000010000 - 0000011000] PGTABLE ==> [0000010000 - 0000011000]
[ 0.000000] #5 [0000011000 - 0000015000] BOOTMAP ==> [0000011000 - 0000015000]
[ 0.000000] Zone PFN ranges:
[ 0.000000] DMA 0x00000010 -> 0x00001000
[ 0.000000] Normal 0x00001000 -> 0x00020000
[ 0.000000] Movable zone start PFN for each node
[ 0.000000] early_node_map[2] active PFN ranges
[ 0.000000] 0: 0x00000010 -> 0x0000009f
[ 0.000000] 0: 0x00000100 -> 0x00020000
[ 0.000000] On node 0 totalpages: 130959
[ 0.000000] free_area_init_node: node 0, pgdat c0395f60, node_mem_map c1000200
[ 0.000000] DMA zone: 32 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 3951 pages, LIFO batch:0
[ 0.000000] Normal zone: 992 pages used for memmap
[ 0.000000] Normal zone: 125984 pages, LIFO batch:31
[ 0.000000] Local APIC disabled by BIOS -- you can enable it with "lapic"
[ 0.000000] Allocating PCI resources starting at 30000000 (gap: 20000000:dff80000)
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 129935
[ 0.000000] Kernel command line: console=ttyS0,19200 root=/dev/nfs ide0=noprobe ide1=noprobe nfsroot=/export/netboot/correlator_pvt/%s,tcp ip=::::::dhcp pci=biosirq BOOT_IMAGE=bzImage-labslcor1.correlator.pvt auto
[ 0.000000] Enabling fast FPU save and restore... done.
[ 0.000000] Enabling unmasked SIMD FPU exception support... done.
[ 0.000000] Initializing CPU#0
[ 0.000000] CPU 0 irqstacks, hard=c03df000 soft=c03de000
[ 0.000000] PID hash table entries: 2048 (order: 11, 8192 bytes)
[ 0.000000] Fast TSC calibration using PIT
[ 0.000000] Detected 1195.647 MHz processor.
[ 0.004000] Console: colour VGA+ 80x25
[ 0.004000] console [ttyS0] enabled
[ 0.004000] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[ 0.004000] ... MAX_LOCKDEP_SUBCLASSES: 8
[ 0.004000] ... MAX_LOCK_DEPTH: 48
[ 0.004000] ... MAX_LOCKDEP_KEYS: 8191
[ 0.004000] ... CLASSHASH_SIZE: 4096
[ 0.004000] ... MAX_LOCKDEP_ENTRIES: 8192
[ 0.004000] ... MAX_LOCKDEP_CHAINS: 16384
[ 0.004000] ... CHAINHASH_SIZE: 8192
[ 0.004000] memory used by lock dependency info: 2335 kB
[ 0.004000] per task-struct memory footprint: 1152 bytes
[ 0.004000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.004000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[ 0.004000] Memory: 512108k/524288k available (1795k kernel code, 11512k reserved, 890k data, 232k init, 0k highmem)
[ 0.004000] virtual kernel memory layout:
[ 0.004000] fixmap : 0xfffac000 - 0xfffff000 ( 332 kB)
[ 0.004000] vmalloc : 0xe0800000 - 0xfffaa000 ( 503 MB)
[ 0.004000] lowmem : 0xc0000000 - 0xe0000000 ( 512 MB)
[ 0.004000] .init : 0xc03a1000 - 0xc03db000 ( 232 kB)
[ 0.004000] .data : 0xc02c0cee - 0xc039f7c8 ( 890 kB)
[ 0.004000] .text : 0xc0100000 - 0xc02c0cee (1795 kB)
[ 0.004000] Checking if this processor honours the WP bit even in supervisor mode...Ok.
[ 0.004000] SLUB: Genslabs=12, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.004025] Calibrating delay loop (skipped), value calculated using timer frequency.. 2391.29 BogoMIPS (lpj=4782588)
[ 0.012060] Mount-cache hash table entries: 512
[ 0.017091] CPU: L1 I cache: 16K, L1 D cache: 16K
[ 0.024006] CPU: L2 cache: 512K
[ 0.028010] Intel machine check architecture supported.
[ 0.032008] Intel machine check reporting enabled on CPU#0.
[ 0.036015] CPU: Intel(R) Pentium(R) III Mobile CPU 1200MHz stepping 01
[ 0.048011] Checking 'hlt' instruction... OK.
[ 0.070988] Freeing SMP alternatives: 0k freed
[ 0.072007] ACPI: Core revision 20080926
[ 0.076077] ACPI Exception (tbxface-0627): AE_NO_ACPI_TABLES, While loading namespace from ACPI tables [20080926]
[ 0.088006] ACPI: Unable to load the System Description Tables
[ 0.093401] net_namespace: 384 bytes
[ 0.096363] NET: Registered protocol family 16
[ 0.105855] PCI: PCI BIOS revision 2.10 entry at 0xfda10, last bus=2
[ 0.108008] PCI: Using configuration type 1 for base access
[ 0.120211] bio: create slab <bio-0> at 0
[ 0.124230] ACPI: Interpreter disabled.
[ 0.132405] usbcore: registered new interface driver usbfs
[ 0.136165] usbcore: registered new interface driver hub
[ 0.140180] usbcore: registered new device driver usb
[ 0.148320] PCI: Probing PCI hardware
[ 0.152051] PCI: Probing PCI hardware (bus 00)
[ 0.152298] * The chipset may have PM-Timer Bug. Due to workarounds for a bug,
[ 0.152302] * this clock source is slow. If you are sure your timer does not have
[ 0.152305] * this bug, please use "acpi_pm_good" to disable the workaround
[ 0.156074] * The chipset may have PM-Timer Bug. Due to workarounds for a bug,
[ 0.156077] * this clock source is slow. If you are sure your timer does not have
[ 0.156080] * this bug, please use "acpi_pm_good" to disable the workaround
[ 0.160120] pci 0000:00:01.0: reg 10 32bit mmio: [0xf9020000-0xf9020fff]
[ 0.160133] pci 0000:00:01.0: reg 14 io port: [0x1080-0x10bf]
[ 0.160145] pci 0000:00:01.0: reg 18 32bit mmio: [0xf9000000-0xf901ffff]
[ 0.160175] pci 0000:00:01.0: reg 30 32bit mmio: [0x000000-0x0fffff]
[ 0.160204] pci 0000:00:01.0: supports D1 D2
[ 0.160209] pci 0000:00:01.0: PME# supported from D0 D1 D2 D3hot
[ 0.164011] pci 0000:00:01.0: PME# disabled
[ 0.168051] pci 0000:00:02.0: reg 10 32bit mmio: [0xf9021000-0xf9021fff]
[ 0.168063] pci 0000:00:02.0: reg 14 io port: [0x10c0-0x10ff]
[ 0.168075] pci 0000:00:02.0: reg 18 32bit mmio: [0xf9040000-0xf905ffff]
[ 0.168105] pci 0000:00:02.0: reg 30 32bit mmio: [0x000000-0x0fffff]
[ 0.168134] pci 0000:00:02.0: supports D1 D2
[ 0.168139] pci 0000:00:02.0: PME# supported from D0 D1 D2 D3hot
[ 0.172010] pci 0000:00:02.0: PME# disabled
[ 0.176062] pci 0000:00:04.0: reg 10 32bit mmio: [0xf9022000-0xf9022fff]
[ 0.176074] pci 0000:00:04.0: reg 14 io port: [0x1000-0x107f]
[ 0.176253] pci 0000:00:0f.1: reg 20 io port: [0x1400-0x140f]
[ 0.176310] pci 0000:00:0f.2: reg 10 32bit mmio: [0xf9023000-0xf9023fff]
[ 0.176491] pci 0000:01:0c.0: reg 10 32bit mmio: [0xf9100000-0xf91fffff]
[ 0.176506] pci 0000:01:0c.0: reg 14 32bit mmio: [0xf9700000-0xf9700fff]
[ 0.176532] pci 0000:01:0c.0: reg 18 64bit mmio: [0xf9600000-0xf96fffff]
[ 0.176556] pci 0000:01:0c.0: reg 20 64bit mmio: [0xf9500000-0xf95fffff]
[ 0.176668] pci 0000:01:0e.0: reg 10 32bit mmio: [0xf9200000-0xf92fffff]
[ 0.176684] pci 0000:01:0e.0: reg 14 32bit mmio: [0xf9701000-0xf9701fff]
[ 0.176708] pci 0000:01:0e.0: reg 18 64bit mmio: [0xf9900000-0xf99fffff]
[ 0.176733] pci 0000:01:0e.0: reg 20 64bit mmio: [0xf9800000-0xf98fffff]
[ 0.176840] pci 0000:00:04.0: bridge 32bit mmio: [0xf9100000-0xf92fffff]
[ 0.176849] pci 0000:00:04.0: bridge 32bit mmio pref: [0xf9500000-0xf99fffff]
[ 0.178609] PCI: Discovered peer bus 02
[ 0.180189] pci 0000:02:03.0: reg 10 32bit mmio: [0xfa000000-0xfaffffff]
[ 0.180235] pci 0000:02:03.0: reg 30 32bit mmio: [0x000000-0x03ffff]
[ 0.181688] pnp: PnP ACPI: disabled
[ 0.185421] pci 0000:00:04.0: PCI bridge, secondary bus 0000:01
[ 0.220873] pci 0000:00:04.0: IO window: disabled
[ 0.250079] pci 0000:00:04.0: MEM window: 0xf9100000-0xf92fffff
[ 0.286569] pci 0000:00:04.0: PREFETCH window: 0x000000f9500000-0x000000f99fffff
[ 0.331918] pci_bus 0000:00: resource 0 io: [0x00-0xffff]
[ 0.331924] pci_bus 0000:00: resource 1 mem: [0x000000-0xffffffff]
[ 0.331929] pci_bus 0000:01: resource 0 mem: [0x0-0x0]
[ 0.331934] pci_bus 0000:01: resource 1 mem: [0xf9100000-0xf92fffff]
[ 0.331940] pci_bus 0000:01: resource 2 mem: [0xf9500000-0xf99fffff]
[ 0.331945] pci_bus 0000:01: resource 3 mem: [0x0-0x0]
[ 0.331950] pci_bus 0000:02: resource 0 io: [0x00-0xffff]
[ 0.331954] pci_bus 0000:02: resource 1 mem: [0x000000-0xffffffff]
[ 0.332010] NET: Registered protocol family 2
[ 0.358447] IP route cache hash table entries: 16384 (order: 4, 65536 bytes)
[ 0.401080] TCP established hash table entries: 65536 (order: 7, 524288 bytes)
[ 0.446213] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes)
[ 0.500128] TCP: Hash tables configured (established 65536 bind 65536)
[ 0.539299] TCP reno registered
[ 0.558459] NET: Registered protocol family 1
[ 0.585465] platform rtc_cmos: registered platform RTC device (no PNP device found)
[ 0.631330] Machine check exception polling timer started.
[ 0.665649] Microcode Update Driver: v2.00 <[email protected]>, Peter Oruba
[ 0.712519] Scanning for low memory corruption every 60 seconds
[ 0.760346] Installing knfsd (copyright (C) 1996 [email protected]).
[ 0.799493] fuse init (API version 7.11)
[ 0.823571] msgmni has been set to 1000
[ 0.847156] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[ 0.891466] io scheduler noop registered
[ 0.914944] io scheduler deadline registered
[ 0.940540] io scheduler cfq registered (default)
[ 0.968910] pci 0000:00:02.0: Firmware left e100 interrupts enabled; disabling
[ 1.012197] pci 0000:02:03.0: Boot video device
[ 1.022963] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[ 1.061391] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 1.098302] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[ 1.135240] e100: Intel(R) PRO/100 Network Driver, 3.5.23-k6-NAPI
[ 1.171721] e100: Copyright(c) 1999-2006 Intel Corporation
[ 1.227398] e100 0000:00:01.0: PME# disabled
[ 1.253387] e100: eth0: e100_probe: addr 0xf9020000, irq 9, MAC addr 00:80:42:12:ad:ff
[ 1.323517] e100 0000:00:02.0: PME# disabled
[ 1.349416] e100: eth1: e100_probe: addr 0xf9021000, irq 11, MAC addr 00:80:42:12:ae:00
[ 1.397462] console [netcon0] enabled
[ 1.419373] netconsole: network logging started
[ 1.446687] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 1.483733] ohci_hcd 0000:00:0f.2: OHCI Host Controller
[ 1.516113] ohci_hcd 0000:00:0f.2: new USB bus registered, assigned bus number 1
[ 1.560431] ohci_hcd 0000:00:0f.2: irq 10, io mem 0xf9023000
[ 1.649658] usb usb1: configuration #1 chosen from 1 choice
[ 1.683403] hub 1-0:1.0: USB hub found
[ 1.705910] hub 1-0:1.0: 4 ports detected
[ 1.731563] PNP: No PS/2 controller found. Probing ports directly.
[ 1.771634] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 1.801376] serio: i8042 AUX port at 0x60,0x64 irq 12
[ 1.832470] mice: PS/2 mouse device common for all mice
[ 1.864545] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[ 1.902681] rtc0: alarms up to one day, 114 bytes nvram
[ 1.934085] cpuidle: using governor ladder
[ 1.958642] cpuidle: using governor menu
[ 1.984820] usbcore: registered new interface driver usbhid
[ 2.018247] usbhid: v2.6:USB HID core driver
[ 2.043923] TCP cubic registered
[ 2.063273] NET: Registered protocol family 17
[ 2.090467] RPC: Registered udp transport module.
[ 2.118648] RPC: Registered tcp transport module.
[ 2.146845] IO APIC resources could be not be allocated.
[ 2.178672] Using IPI Shortcut mode
[ 2.200498] rtc_cmos rtc_cmos: setting system clock to 2009-01-15 10:08:28 UTC (1232014108)
[ 2.768492] e100 0000:00:01.0: firmware: using built-in firmware e100/d101s_ucode.bin
[ 2.835120] e100: eth0 NIC Link is Up 100 Mbps Full Duplex
[ 2.876469] e100 0000:00:02.0: firmware: using built-in firmware e100/d101s_ucode.bin
[ 3.378827] Clocksource tsc unstable (delta = 115900197 ns)
[ 3.970856] Sending DHCP requests ., OK
[ 3.997668] IP-Config: Got DHCP answer from 192.168.5.13, my address is 192.168.17.13
[ 4.005875] IP-Config: Complete:
[ 4.008824] device=eth0, addr=192.168.17.13, mask=255.255.255.0, gw=192.168.17.1,
[ 4.026488] host=192.168.17.13, domain=correlator.pvt fileserver.pvt ovro.pvt mmarray.org carma.pvt pr, nis-domain=(none),
[ 4.028216] bootserver=192.168.5.13, rootserver=192.168.5.13, rootpath=
[ 4.041384] Looking up port of RPC 100003/2 on 192.168.5.13
[ 4.050478] Looking up port of RPC 100005/1 on 192.168.5.13
[ 4.129452] VFS: Mounted root (nfs filesystem) readonly on device 0:13.
[ 4.133518] Freeing unused kernel memory: 232k freed
[ 6.834682] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 6.863495] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4



00:00.0 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
00:00.1 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
00:01.0 Ethernet controller: Intel Corporation 8255xER/82551IT Fast Ethernet Controller (rev 09)
00:02.0 Ethernet controller: Intel Corporation 8255xER/82551IT Fast Ethernet Controller (rev 09)
00:04.0 PCI bridge: Force Computers Device 0001 (rev 01)
00:0f.0 ISA bridge: Broadcom OSB4 South Bridge (rev 50)
00:0f.1 IDE interface: Broadcom OSB4 IDE Controller
00:0f.2 USB Controller: Broadcom OSB4/CSB5 OHCI USB Controller (rev 04)
01:0c.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)
01:0e.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)
02:03.0 VGA compatible controller: Chips and Technologies F69030 (rev 61)



00:00.0 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
Latency: 48, Cache Line Size: 32 bytes

00:00.1 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64, Cache Line Size: 32 bytes

00:01.0 Ethernet controller: Intel Corporation 8255xER/82551IT Fast Ethernet Controller (rev 09)
Subsystem: Intel Corporation Device 100c
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 66 (2000ns min, 14000ns max), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 9
Region 0: Memory at f9020000 (32-bit, non-prefetchable) [size=4K]
Region 1: I/O ports at 1080 [size=64]
Region 2: Memory at f9000000 (32-bit, non-prefetchable) [size=128K]
[virtual] Expansion ROM at 30000000 [disabled] [size=1M]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=2 PME-
Kernel driver in use: e100

00:02.0 Ethernet controller: Intel Corporation 8255xER/82551IT Fast Ethernet Controller (rev 09)
Subsystem: Intel Corporation Device 100c
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 66 (2000ns min, 14000ns max), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 11
Region 0: Memory at f9021000 (32-bit, non-prefetchable) [size=4K]
Region 1: I/O ports at 10c0 [size=64]
Region 2: Memory at f9040000 (32-bit, non-prefetchable) [size=128K]
[virtual] Expansion ROM at 30100000 [disabled] [size=1M]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=2 PME-
Kernel driver in use: e100

00:04.0 PCI bridge: Force Computers Device 0001 (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64, Cache Line Size: 32 bytes
Region 0: Memory at f9022000 (32-bit, non-prefetchable) [size=4K]
Region 1: I/O ports at 1000 [size=128]
Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: f9100000-f92fffff
Prefetchable memory behind bridge: f9500000-f99fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [fc] CompactPCI hot-swap <?>
Capabilities: [f0] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable-
Address: 00000000 Data: 0000
Kernel modules: shpchp

00:0f.0 ISA bridge: Broadcom OSB4 South Bridge (rev 50)
Subsystem: Broadcom OSB4 South Bridge
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0

00:0f.1 IDE interface: Broadcom OSB4 IDE Controller (prog-if 8a [Master SecP PriP])
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64
Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable) [disabled] [size=8]
Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable) [disabled] [size=1]
Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable) [disabled] [size=8]
Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable) [disabled] [size=1]
Region 4: I/O ports at 1400 [size=16]

00:0f.2 USB Controller: Broadcom OSB4/CSB5 OHCI USB Controller (rev 04) (prog-if 10 [OHCI])
Subsystem: Broadcom OSB4/CSB5 OHCI USB Controller
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64 (20000ns max), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 10
Region 0: Memory at f9023000 (32-bit, non-prefetchable) [size=4K]
Kernel driver in use: ohci_hcd

01:0c.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 10
Region 0: Memory at f9100000 (32-bit, non-prefetchable) [size=1M]
Region 1: Memory at f9700000 (32-bit, prefetchable) [size=4K]
Region 2: Memory at f9600000 (64-bit, prefetchable) [size=1M]
Region 4: Memory at f9500000 (64-bit, prefetchable) [size=1M]
Capabilities: [48] CompactPCI hot-swap <?>

01:0e.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 11
Region 0: Memory at f9200000 (32-bit, non-prefetchable) [size=1M]
Region 1: Memory at f9701000 (32-bit, prefetchable) [size=4K]
Region 2: Memory at f9900000 (64-bit, prefetchable) [size=1M]
Region 4: Memory at f9800000 (64-bit, prefetchable) [size=1M]
Capabilities: [48] CompactPCI hot-swap <?>

02:03.0 VGA compatible controller: Chips and Technologies F69030 (rev 61) (prog-if 00 [VGA controller])
Subsystem: Chips and Technologies F69030
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping+ SERR+ FastB2B- DisINTx-
Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 9
Region 0: Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
[virtual] Expansion ROM at 30200000 [disabled] [size=256K]

2009-01-15 19:21:58

by Ira W. Snyder

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Thu, Jan 15, 2009 at 06:53:51PM +0100, Arnd Bergmann wrote:
> On Thursday 15 January 2009, Ira Snyder wrote:
> >
> > These are PCI boards, not PCIe. The host computers are all Pentium3-M
> > systems. I tried enabling MSI on the Freescale boards in the driver, by
> > calling pci_enable_msi() during probe(), and it failed.
>
> That doesn't really mean anything, just that the PCI endpoint doesn't
> announce its capability to do MSI in the config space, or that it
> does not have an interrupt line. Since you basically implement the
> device on the FSL board, you should also be able to define the interrupt
> capabilities by writing to the config space.
>
> Do you know what kind of chipset the host uses? It should be fairly
> simple to find out whether or not it can do MSI.
>

I have another question for you Arnd.

What did you use as the host and guest drivers when you ran virtio over
PCI? Did you use two unmodified instances of virtio_net (one on host,
one on guest) for networking, or did you write new virtio drivers for
those? How about for virtio_console (if you ran it at all).

Thanks again,
Ira

2009-01-15 21:08:25

by Arnd Bergmann

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Thursday 15 January 2009, Ira Snyder wrote:
> Some sort of Broadcom chipset, I think. Full dmesg and lspci output are
> appended.
>
> The PCI bridge does mention MSI, so maybe it does support it. Would
> using the DMA from the host mean that the guest system couldn't use the
> DMA controller at all? All of the channels share the same interrupt line
> on the guest. I need one channel on the DMA controller on the guest to
> do realtime transfers from some data processing FPGAs on the board.

The PCI-CompactPCI bridge supports MSI, but that does not imply that
the host does. My limited understanding of x86 tells me that you need
to enable the I/O-APIC in order to use MSI. Your log shows two relavant
lines:

[ 0.000000] Local APIC disabled by BIOS -- you can enable it with "lapic"
and
[ 2.146845] IO APIC resources could be not be allocated.

I think you first need to enable the local APIC, then find out what the
problem with the IO APIC is. If possible, try enabling ACPI (sic) in the
BIOS in order to get the APIC support.

If you get that to work, you can use the interrupt line of the DMA controller
to signal interrupts to the FSL machine, while using DMA transfers to the
MSI address for sending interrupts to the host machine.

Arnd <><

2009-01-15 21:24:15

by Arnd Bergmann

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Thursday 15 January 2009, Ira Snyder wrote:
> I have another question for you Arnd.
>
> What did you use as the host and guest drivers when you ran virtio over
> PCI? Did you use two unmodified instances of virtio_net (one on host,
> one on guest) for networking, or did you write new virtio drivers for
> those? How about for virtio_console (if you ran it at all).

Jan-Bernd may be able to tell you details about this, and send you the
driver code that his interns implemented for it.
This was only doing virtio_net between two machines using MMIO transfers,
i.e. the DMA engine was unused, but there was a mailbox interrupt (if you
have one of these, you won't need MSI, btw -- just write a DMA to it).

Arnd <><

2009-01-15 21:41:19

by Ira W. Snyder

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Thu, Jan 15, 2009 at 10:22:53PM +0100, Arnd Bergmann wrote:
> On Thursday 15 January 2009, Ira Snyder wrote:
> > I have another question for you Arnd.
> >
> > What did you use as the host and guest drivers when you ran virtio over
> > PCI? Did you use two unmodified instances of virtio_net (one on host,
> > one on guest) for networking, or did you write new virtio drivers for
> > those? How about for virtio_console (if you ran it at all).
>
> Jan-Bernd may be able to tell you details about this, and send you the
> driver code that his interns implemented for it.
> This was only doing virtio_net between two machines using MMIO transfers,
> i.e. the DMA engine was unused, but there was a mailbox interrupt (if you
> have one of these, you won't need MSI, btw -- just write a DMA to it).
>

Thanks. Jan-Bernd, I'm looking forward to any input you have.

I'd be happy enough to use mmio at this point. My early attempts with
this driver only used mmio as well. The DMA was bolted on later.

I do have mailboxes (two inbound, two outbound) which can generate
interrupts, as well as doorbell registers (one inbound, one outbound).
The doorbell register's bits are "write 1 to clear", and can only be
cleared by the opposite side.

All of them can cause interrupts over PCI. I used the doorbell registers
to communicate which action needed to be taken in my driver. One
doorbell for "receive packet(s)", another for "packet transmission(s)
complete", etc. I used the mailboxes to transfer characters for the
virtual serial port.

Ira

2009-01-15 23:02:39

by Arnd Bergmann

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Thursday 15 January 2009, Ira Snyder wrote:
> I do have mailboxes (two inbound, two outbound) which can generate
> interrupts, as well as doorbell registers (one inbound, one outbound).
> The doorbell register's bits are "write 1 to clear", and can only be
> cleared by the opposite side.
>

Ok, in this case, you should be able to use one of these to implement
the DMA complete interrupt, but adding an extra DMA transfer to the
mailbox after each transfer that you want to get notified for.

Arnd <><

2009-01-15 23:28:24

by Ira W. Snyder

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Thu, Jan 15, 2009 at 09:57:41PM +0100, Arnd Bergmann wrote:
> On Thursday 15 January 2009, Ira Snyder wrote:
> > Some sort of Broadcom chipset, I think. Full dmesg and lspci output are
> > appended.
> >
> > The PCI bridge does mention MSI, so maybe it does support it. Would
> > using the DMA from the host mean that the guest system couldn't use the
> > DMA controller at all? All of the channels share the same interrupt line
> > on the guest. I need one channel on the DMA controller on the guest to
> > do realtime transfers from some data processing FPGAs on the board.
>
> The PCI-CompactPCI bridge supports MSI, but that does not imply that
> the host does. My limited understanding of x86 tells me that you need
> to enable the I/O-APIC in order to use MSI. Your log shows two relavant
> lines:
>
> [ 0.000000] Local APIC disabled by BIOS -- you can enable it with "lapic"
> and
> [ 2.146845] IO APIC resources could be not be allocated.
>
> I think you first need to enable the local APIC, then find out what the
> problem with the IO APIC is. If possible, try enabling ACPI (sic) in the
> BIOS in order to get the APIC support.
>
> If you get that to work, you can use the interrupt line of the DMA controller
> to signal interrupts to the FSL machine, while using DMA transfers to the
> MSI address for sending interrupts to the host machine.
>

I managed to get the local APIC enabled by adding "lapic" to the kernel
command line. It didn't seem to change anything.

I am totally unable to get the IO APIC working. I hunch that these
boards have totally broken ACPI support, judging from the kernel log.
The BIOS doesn't have much in the way of settings, but I tried the ones
that made sense. No luck there.

Oh well. As in our further discussion, the mailbox or doorbell registers
could be used to trigger DMA interrupts without problems.

Ira

2009-01-15 23:31:58

by Ira W. Snyder

Subject: Re: [PATCH RFC v5] net: add PCINet driver

On Thu, Jan 15, 2009 at 11:53:24PM +0100, Arnd Bergmann wrote:
> On Thursday 15 January 2009, Ira Snyder wrote:
> > I do have mailboxes (two inbound, two outbound) which can generate
> > interrupts, as well as doorbell registers (one inbound, one outbound).
> > The doorbell register's bits are "write 1 to clear", and can only be
> > cleared by the opposite side.
> >
>
> Ok, in this case, you should be able to use one of these to implement
> the DMA complete interrupt, but adding an extra DMA transfer to the
> mailbox after each transfer that you want to get notified for.

Yep. I wouldn't have come up with that idea on my own, but it seems
obvious now :)

It seems a little strange to keep the DMA descriptors host memory (the
DMA controller will need to issue a PCI read to get them) but I guess
that is how real network cards work.

Thanks,
Ira