Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760031AbbLCLRU (ORCPT ); Thu, 3 Dec 2015 06:17:20 -0500 Received: from mail-bn1on0074.outbound.protection.outlook.com ([157.56.110.74]:35354 "EHLO na01-bn1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1759975AbbLCLLc (ORCPT ); Thu, 3 Dec 2015 06:11:32 -0500 Authentication-Results: spf=none (sender IP is 192.88.168.50) smtp.helo=tx30smr01.am.freescale.net; freescale.mail.onmicrosoft.com; dkim=none (message not signed) header.d=none;freescale.mail.onmicrosoft.com; dmarc=none action=none header.from=; From: <> To: CC: , , , , , , , , , , Madalin Bucur Subject: [net-next v5 3/8] dpaa_eth: add support for S/G frames Date: Thu, 3 Dec 2015 14:09:01 +0200 Message-ID: <1449144546-25438-4-git-send-email-madalin.bucur@freescale.com> X-Mailer: git-send-email 1.5.6.5 In-Reply-To: <1449144546-25438-1-git-send-email-madalin.bucur@freescale.com> References: <1449144546-25438-1-git-send-email-madalin.bucur@freescale.com> Reply-To: X-EOPAttributedMessage: 0 X-Matching-Connectors: 130936146903985359;(91ab9b29-cfa4-454e-5278-08d120cd25b8);() X-Microsoft-Exchange-Diagnostics: 1;BY2FFO11OLC014;1:ZEwCRojndoe+giwpRy6CXvVtRHhbFx+ZWBqHDfWTo/YFyDS6vb4QW7uhsvS2RA0Xn4Fxot0RZrPVRZFppqE1qXHq13AGmGDRo+FElNI/DP+MXSt82a3fza2NBO8WvPsgHXG+JxABGlGcSyOAUrR1J9qB2gqXBx9Wbi1KbgokywsF4kCS3XWABqIiKYxccvmCgMLzPQhQY5041Vy3JRXtx7QbkcIPVpl0XeuvJAGfB45+kh+eVAl/fBXnSOYm4/m7mtrGWZGJN5qazYis1zkOBfz7wYM5fdmLTk2hS9Mt8XsHDFXzpBEsd8y+YHowp+ekY/gEupkwxsFO+ooY2SBSxoQHwkymOdPzEp/SvBkof9r1l2vrgTeS0881yho84OgZCI7lp05b8WIOTCZ2i41T9Bsu8ipZbSTec2hFA1nmY10pAZgvXTiQ4MDfQaRRE8f0Vf6nKSxOvjuwlf+ioSWy8A== X-Forefront-Antispam-Report: CIP:192.88.168.50;CTRY:US;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(10009020)(979002)(6009001)(2970300002)(428002)(189002)(199003)(6806005)(47776003)(11100500001)(2351001)(87936001)(33646002)(49486002)(50986999)(5008740100001)(2950100001)(1096002)(48376002)(77096005)(229853001)(104016004)(50466002)(1220700001)(586003)(4001430100002)(19580395003)(97736004)(189998001)(81156007)(36756003)(78992003)(107886002)(110136002)(19580405001)(5003940100001)(5001960100002)(5005630100001)(105586002)(106466001)(43066003)(101416001)(76176999)(53806999)(50226001)(60522002)(969003)(989001)(999001)(1009001)(1019001);DIR:OUT;SFP:1101;SCL:1;SRVR:BLUPR0301MB2114;H:tx30smr01.am.freescale.net;FPR:;SPF:None;PTR:InfoDomainNonexistent;MX:0;A:0;LANG:en; MIME-Version: 1.0 Content-Type: text/plain X-Microsoft-Exchange-Diagnostics: 1;BLUPR0301MB2114;2:5vIkpWUBne1KqgqGMx8MfNkRxGyldkReDNxiHw7I0Tadjwd1dEOd1szlZKyFaJZXO+wJcbJrB1mMHgyEYmtzOe/4yUM0j8A3YG8djq3J/2lJUab/ee2rYMvK/lRxS5UwtNNFeB6Kf85fuPLZXYIZvw==;3:N7RauZ5zq5mMqZomC7K6o92lLr0pZUCHCdaJh2b8qzDEbnwVq9dBqn748xzN5CXLcvJCUXhJkbhJPecKaplQ4nGuHf6p/JeNRBnechYSYh8XvMYUG3fcQhnG6J45fcwMm8BJO/KMuK96AmpW8mJi+z/YanoCYkpiZN0g3mHVZfly88sT3oW2aRwoB3+DAQD04DBTkvl4tk1W/kzrtx98dNYQDZ+2FYpLaOjwTA9kP2c=;25:WlKmYQQXLlqkw8tfS10g3HonCHGOXXGuiWMxPgDMDPz+MC0x0YHDE1w4d1kKXOxSQRAUyTGx9v7a7ZodraLRQ/n9IwsbyLnyzWzJ3POqzc8u+NE+HeubNehC79rwae42CNyJ76aRD964Z65y/LjS2/JQwiyK5aOzfw1z4GaXR0nfIn0yJfTeXqtMOWNR0e9sojhWDA6WPpoP+DYOKRf8H7zgQbk++OjXrYWQhM8JzXw3qRA5zJuIyTqA3iSKrdf+fuvErHugF5gjvpVggOYfHQ== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BLUPR0301MB2114; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(101931422205132); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(8121501046)(520078)(5005006)(10201501046)(3002001);SRVR:BLUPR0301MB2114;BCL:0;PCL:0;RULEID:(400006);SRVR:BLUPR0301MB2114; X-Microsoft-Exchange-Diagnostics: 1;BLUPR0301MB2114;4:G6OwqIv1FP56q2Gt/NkepuEdnLmtElSithp/DSlF/TOt2j+c3xVQ2Vi0xzfQbu0Z3RkzMJqj7R7Q8IKd+tUkWUJYMC5owZiDmbYCzM0IqkrVUKpoX/uKeVOodKHH4TLMWyvdSsGo0E2dMCO3+YCV2dEhlRVb4epkSkP5cJ1MApIlu92GvDKmNfQt6JwYjZP2Hj0InTMnEyPtkoCZ+T+YFcXVI+CAcpGdT757UMHQVLt6kMx/FUiaFFcqRdsApJwUqnkYYPCLXvFknDT4B9icsf8LiNwmNIh/uWypHsRGF2TH6Wr5/dOdXbL95Aj1bOl2JmBRJSIFxjP7lreao/UBv9p3Af1YJagv5nxbFjfrzDcV2HE3D4NcBm8rc5CJW0oy3xpfM9wSgJkLxXk8ZALk7Ci1Y6d7yn3nWX6jc8LJ2/cGNxDKzLBtPHgW4GJePAeD X-Forefront-PRVS: 077929D941 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;BLUPR0301MB2114;23:9NvwXNh+sFViarv7HvpR4mwKco1dTXk4caWFhC5?= =?us-ascii?Q?dz+QEMNeX2DX1vGKnWxKVW/LzBARzUarGrE9sRkuvqioLwDL2EQFVsxPV3F+?= =?us-ascii?Q?53F/7PvcTdFCovItpuFRndbHcoTlwKqgTQrVPiiKjhlcEdBm5DHPujkAcdYV?= =?us-ascii?Q?FceruGT3kq4kS+QEgeXqqOkG1F+GKsPtVwHdK7z6uwu9J/G0UEvEKyzTpOPx?= =?us-ascii?Q?r5RGgc11+Nzcax/gUMCaLTcahnrPSbP7/PZS1Fs3SmfBbP80NPEHHJzt8GQm?= =?us-ascii?Q?Z9lntIivPm0cqXmsrs6HX/vjFGA3NsO3leNxSsxiWIGAbE4y3X4/+zsPFVMk?= =?us-ascii?Q?sftKlR364N5zEpzeRnZsxzTuQSiYrO9MTyqzSjKnExPHAJUs7cxGmvMrnWnn?= =?us-ascii?Q?jtIgjJuUXyRkiQPSAB2tIwDAN0Txvo+MrS+7ckHK5OnmAxdfuDJwUJwlCP9+?= =?us-ascii?Q?48oVSHo+ZTE537ADeBvfIrKjiMPmwdJJ0p4wiOTCh5SML28laqrB+zTy/tjO?= =?us-ascii?Q?/K90R3Doh2taYpNpA61/jeI8dSEWZ003nSC/D/joOmHxUQCQ8yylqjLKBNbO?= =?us-ascii?Q?bxF86ibDNkO+BEx6BvxTK9tjxmjUT1o9MRQ7wNPC0AW9fMH+SVHjvF/3nNW4?= =?us-ascii?Q?yGXfBDCdNHypbWHJSlXbPGncRtXlul9IAfnPC4dvy6OsZVF8UCWk9eDLj9ia?= =?us-ascii?Q?0Rh0T3YA+YEiRGjZvz5HjaxJZVOY0ZRbrUMphiMqsKWcUY6H4HfDRxWV2pfk?= =?us-ascii?Q?XnUmVM5jxy8brhpNfaqMAgqVERJYOmR6ZExzKMDux9FL9rpwaEZQVYCdOwgV?= =?us-ascii?Q?aKAXW33npltl7Agq1cAztFbHVOgAPP3KeLUmgDXimi+yUzxpOxv/2WP+ZKl1?= =?us-ascii?Q?gXeiNMSeWaIUjwVmZsJxgCMoshNw0itvqmtOinwHb6c0szo/3ADKYo0469eY?= =?us-ascii?Q?5kjnoFPtg2V3yx+2+99F7/4U2vgWRYcmp4C+3zeyHMrkElzTrg3w4dnbE//1?= =?us-ascii?Q?YpxSnSuEFCNcaZE9YZEV4j2YdTCeZNQcSFO2+uRs9H3qYzEEL9aSN0ezoMNH?= =?us-ascii?Q?KMBoVxWikL/doH8+xUfGKOocpsDPLHJmrbPt6aj5TZ65bZzfJx4keMnoOTeR?= =?us-ascii?Q?gUPzyEBgIo1uuZcHAzXhX5sDnyqEquP5x+80uUAcdvQN0vt4Xfe/cmrBLpbz?= =?us-ascii?Q?tRBXNMmc0aAzwEyNr3Jsdh4tsWkQqwLBZI8+W2KlZ74nO/rzF3LwLAXtOqRY?= =?us-ascii?Q?rCPy2OgfqrM7CSO0Ex8/J1Nbsbj2MBKepmsGyWGly?= X-Microsoft-Exchange-Diagnostics: 1;BLUPR0301MB2114;5:48b5RY6oiOb/KxwX9p8BMYYi6NDUHhCe0VE7GbFHMc1h9LXEcnX3gnad3Oc0ip/tgoYy6EBwWEIHVbwRpMI4UZn2QH9+TbLpORFarP+aiGgkgEDOk941LpW6xhqnxxFjRSzt//jmBrBR5OlLoLZ4f9mwyq84IixA9Y7AQYWx6w4=;24:GzG6dKvIzQLVKTY7NzxccWpFOpbFLW/LjATIEc2FqQHzRnEcl6Kt5Q6eNe4TRxLYZ4jOUDCuQlaNcWEuLgqUh5+4ybmWwo685DIgS6SQZGw= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Dec 2015 11:11:30.1957 (UTC) X-MS-Exchange-CrossTenant-Id: 5afe0b00-7697-4969-b663-5eab37d5f47e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=5afe0b00-7697-4969-b663-5eab37d5f47e;Ip=[192.88.168.50];Helo=[tx30smr01.am.freescale.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR0301MB2114 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 16763 Lines: 527 From: Madalin Bucur Add support for Scatter/Gather (S/G) frames. The FMan can place the frame content into multiple buffers and provide a S/G Table (SGT) into one first buffer with references to the others. Signed-off-by: Madalin Bucur --- .../net/ethernet/freescale/dpaa/dpaa_eth_common.c | 54 +++- .../net/ethernet/freescale/dpaa/dpaa_eth_common.h | 2 + drivers/net/ethernet/freescale/dpaa/dpaa_eth_sg.c | 337 +++++++++++++++++++-- 3 files changed, 373 insertions(+), 20 deletions(-) diff --git a/drivers/net/ethernet/freescale/dpaa/dpaa_eth_common.c b/drivers/net/ethernet/freescale/dpaa/dpaa_eth_common.c index c96995c..1c82338 100644 --- a/drivers/net/ethernet/freescale/dpaa/dpaa_eth_common.c +++ b/drivers/net/ethernet/freescale/dpaa/dpaa_eth_common.c @@ -117,6 +117,10 @@ int dpa_netdev_init(struct net_device *net_dev, net_dev->hw_features |= (NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM | NETIF_F_LLTX); + net_dev->hw_features |= NETIF_F_SG | NETIF_F_HIGHDMA; + /* The kernels enables GSO automatically, if we declare NETIF_F_SG. + * For conformity, we'll still declare GSO explicitly. + */ net_dev->features |= NETIF_F_GSO; net_dev->priv_flags |= IFF_LIVE_ADDR_CHANGE; @@ -1194,10 +1198,42 @@ void dpaa_eth_init_ports(struct mac_device *mac_dev, port_fqs->rx_defq, &buf_layout[RX]); } +void dpa_release_sgt(struct qm_sg_entry *sgt) +{ + struct dpa_bp *dpa_bp; + struct bm_buffer bmb[DPA_BUFF_RELEASE_MAX]; + int i = 0, j, timeout = 100; + + memset(bmb, 0, sizeof(bmb)); + + do { + dpa_bp = dpa_bpid2pool(sgt[i].bpid); + WARN_ON(!dpa_bp); + + j = 0; + do { + WARN_ON(sgt[i].extension); + + bmb[j].hi = sgt[i].addr_hi; + bmb[j].lo = be32_to_cpu(sgt[i].addr_lo); + + j++; i++; + } while (j < ARRAY_SIZE(bmb) && + !sgt[i - 1].final && + sgt[i - 1].bpid == sgt[i].bpid); + + while (bman_release(dpa_bp->pool, bmb, j, 0) && --timeout) + cpu_relax(); + } while (!sgt[i - 1].final); +} + void dpa_fd_release(const struct net_device *net_dev, const struct qm_fd *fd) { + struct qm_sg_entry *sgt; struct dpa_bp *dpa_bp; struct bm_buffer bmb; + dma_addr_t addr; + void *vaddr; int timeout = 100; memset(&bmb, 0, sizeof(bmb)); @@ -1206,7 +1242,23 @@ void dpa_fd_release(const struct net_device *net_dev, const struct qm_fd *fd) dpa_bp = dpa_bpid2pool(fd->bpid); WARN_ON(!dpa_bp); - WARN_ON(fd->format == qm_fd_sg); + if (fd->format == qm_fd_sg) { + vaddr = phys_to_virt(fd->addr); + sgt = vaddr + dpa_fd_offset(fd); + + dma_unmap_single(dpa_bp->dev, qm_fd_addr(fd), dpa_bp->size, + DMA_BIDIRECTIONAL); + + dpa_release_sgt(sgt); + + addr = dma_map_single(dpa_bp->dev, vaddr, dpa_bp->size, + DMA_BIDIRECTIONAL); + if (dma_mapping_error(dpa_bp->dev, addr)) { + dev_err(dpa_bp->dev, "DMA mapping failed"); + return; + } + bm_buffer_set64(&bmb, addr); + } while (bman_release(dpa_bp->pool, &bmb, 1, 0) && --timeout) cpu_relax(); diff --git a/drivers/net/ethernet/freescale/dpaa/dpaa_eth_common.h b/drivers/net/ethernet/freescale/dpaa/dpaa_eth_common.h index 78a97d9..c3443e6 100644 --- a/drivers/net/ethernet/freescale/dpaa/dpaa_eth_common.h +++ b/drivers/net/ethernet/freescale/dpaa/dpaa_eth_common.h @@ -37,6 +37,7 @@ #include "dpaa_eth.h" +#define DPA_SGT_MAX_ENTRIES 16 /* maximum number of entries in SG Table */ #define DPA_BUFF_RELEASE_MAX 8 /* maximum number of buffers released at once */ /* used in napi related functions */ @@ -89,6 +90,7 @@ void dpaa_eth_init_ports(struct mac_device *mac_dev, struct fm_port_fqs *port_fqs, struct dpa_buffer_layout *buf_layout, struct device *dev); +void dpa_release_sgt(struct qm_sg_entry *sgt); void dpa_fd_release(const struct net_device *net_dev, const struct qm_fd *fd); int dpa_enable_tx_csum(struct dpa_priv *priv, struct sk_buff *skb, diff --git a/drivers/net/ethernet/freescale/dpaa/dpaa_eth_sg.c b/drivers/net/ethernet/freescale/dpaa/dpaa_eth_sg.c index c913dd6..5fbc167 100644 --- a/drivers/net/ethernet/freescale/dpaa/dpaa_eth_sg.c +++ b/drivers/net/ethernet/freescale/dpaa/dpaa_eth_sg.c @@ -38,6 +38,32 @@ #include "dpaa_eth.h" #include "dpaa_eth_common.h" +/* DMA map and add a page frag back into the bpool. + * @vaddr fragment must have been allocated with netdev_alloc_frag(), + * specifically for fitting into @dpa_bp. + */ +static void dpa_bp_recycle_frag(struct dpa_bp *dpa_bp, unsigned long vaddr, + int *count_ptr) +{ + struct bm_buffer bmb; + dma_addr_t addr; + int timeout = 100; + + addr = dma_map_single(dpa_bp->dev, (void *)vaddr, dpa_bp->size, + DMA_BIDIRECTIONAL); + if (unlikely(dma_mapping_error(dpa_bp->dev, addr))) { + dev_err(dpa_bp->dev, "DMA mapping failed"); + return; + } + + bm_buffer_set64(&bmb, addr); + + while (bman_release(dpa_bp->pool, &bmb, 1, 0) && --timeout) + cpu_relax(); + + (*count_ptr)++; +} + static int dpa_bp_add_8_bufs(const struct dpa_bp *dpa_bp) { struct bm_buffer bmb[8]; @@ -168,14 +194,45 @@ int dpaa_eth_refill_bpools(struct dpa_bp *dpa_bp, int *countptr) struct sk_buff *dpa_cleanup_tx_fd(const struct dpa_priv *priv, const struct qm_fd *fd) { + const struct qm_sg_entry *sgt; + int i; struct dpa_bp *dpa_bp = priv->dpa_bp; dma_addr_t addr = qm_fd_addr(fd); struct sk_buff **skbh = (struct sk_buff **)phys_to_virt(addr); struct sk_buff *skb = *skbh; const enum dma_data_direction dma_dir = DMA_TO_DEVICE; + int nr_frags; + + if (unlikely(fd->format == qm_fd_sg)) { + nr_frags = skb_shinfo(skb)->nr_frags; + dma_unmap_single(dpa_bp->dev, addr, dpa_fd_offset(fd) + + sizeof(struct qm_sg_entry) * (1 + nr_frags), + dma_dir); + + /* The sgt buffer has been allocated with netdev_alloc_frag(), + * it's from lowmem. + */ + sgt = phys_to_virt(addr + dpa_fd_offset(fd)); + + /* sgt[0] is from lowmem, was dma_map_single()-ed */ + dma_unmap_single(dpa_bp->dev, (dma_addr_t)sgt[0].addr, + sgt[0].length, dma_dir); + + /* remaining pages were mapped with dma_map_page() */ + for (i = 1; i < nr_frags; i++) { + WARN_ON(sgt[i].extension); + + dma_unmap_page(dpa_bp->dev, (dma_addr_t)sgt[i].addr, + sgt[i].length, dma_dir); + } + + /* Free the page frag that we allocated on Tx */ + put_page(virt_to_head_page(sgt)); + } else { + dma_unmap_single(dpa_bp->dev, addr, + skb_tail_pointer(skb) - (u8 *)skbh, dma_dir); + } - dma_unmap_single(dpa_bp->dev, addr, - skb_tail_pointer(skb) - (u8 *)skbh, dma_dir); return skb; } @@ -210,6 +267,109 @@ static struct sk_buff *contig_fd_to_skb(const struct dpa_priv *priv, return skb; } +/* Build an skb with the data of the first S/G entry in the linear portion and + * the rest of the frame as skb fragments. + * + * The page fragment holding the S/G Table is recycled here. + */ +static struct sk_buff *sg_fd_to_skb(const struct dpa_priv *priv, + const struct qm_fd *fd, + int *count_ptr) +{ + const struct qm_sg_entry *sgt; + dma_addr_t addr = qm_fd_addr(fd); + ssize_t fd_off = dpa_fd_offset(fd); + dma_addr_t sg_addr; + void *vaddr, *sg_vaddr; + struct dpa_bp *dpa_bp; + struct page *page, *head_page; + int frag_offset, frag_len; + int page_offset; + int i; + struct sk_buff *skb = NULL, *skb_tmp, **skbh; + + vaddr = phys_to_virt(addr); + WARN_ON(!IS_ALIGNED((unsigned long)vaddr, SMP_CACHE_BYTES)); + + dpa_bp = priv->dpa_bp; + /* Iterate through the SGT entries and add data buffers to the skb */ + sgt = vaddr + fd_off; + for (i = 0; i < DPA_SGT_MAX_ENTRIES; i++) { + /* Extension bit is not supported */ + WARN_ON(sgt[i].extension); + + /* We use a single global Rx pool */ + WARN_ON(dpa_bp != dpa_bpid2pool(sgt[i].bpid)); + + sg_addr = qm_sg_addr(&sgt[i]); + sg_vaddr = phys_to_virt(sg_addr); + WARN_ON(!IS_ALIGNED((unsigned long)sg_vaddr, + SMP_CACHE_BYTES)); + + dma_unmap_single(dpa_bp->dev, sg_addr, dpa_bp->size, + DMA_BIDIRECTIONAL); + if (i == 0) { + skbh = (struct sk_buff **)sg_vaddr; + skb = *(skbh - 1); + WARN_ON(skb->head != sg_vaddr); + + skb->ip_summed = CHECKSUM_NONE; + + /* Make sure forwarded skbs will have enough space + * on Tx, if extra headers are added. + */ + WARN_ON(fd_off != priv->rx_headroom); + skb_reserve(skb, fd_off); + skb_put(skb, sgt[i].length); + } else { + /* Not the first S/G entry; all data from buffer will + * be added in an skb fragment; fragment index is offset + * by one since first S/G entry was incorporated in the + * linear part of the skb. + * + * Caution: 'page' may be a tail page. + */ + skbh = (struct sk_buff **)sg_vaddr; + skb_tmp = *(skbh - 1); + page = virt_to_page(sg_vaddr); + head_page = virt_to_head_page(sg_vaddr); + + /* Free (only) the skbuff shell because its data buffer + * is already a frag in the main skb. + */ + get_page(head_page); + dev_kfree_skb(skb_tmp); + + /* Compute offset in (possibly tail) page */ + page_offset = ((unsigned long)sg_vaddr & + (PAGE_SIZE - 1)) + + (page_address(page) - page_address(head_page)); + /* page_offset only refers to the beginning of sgt[i]; + * but the buffer itself may have an internal offset. + */ + frag_offset = sgt[i].offset + page_offset; + frag_len = sgt[i].length; + /* skb_add_rx_frag() does no checking on the page; if + * we pass it a tail page, we'll end up with + * bad page accounting and eventually with segafults. + */ + skb_add_rx_frag(skb, i - 1, head_page, frag_offset, + frag_len, dpa_bp->size); + } + /* Update the pool count for the current {cpu x bpool} */ + (*count_ptr)--; + + if (sgt[i].final) + break; + } + WARN_ONCE(i == DPA_SGT_MAX_ENTRIES, "No final bit on SGT\n"); + + /* recycle the SGT fragment */ + WARN_ON(dpa_bp != dpa_bpid2pool(fd->bpid)); + dpa_bp_recycle_frag(dpa_bp, (unsigned long)vaddr, count_ptr); + return skb; +} + void dpa_rx(struct net_device *net_dev, struct qman_portal *portal, const struct dpa_priv *priv, @@ -237,17 +397,20 @@ void dpa_rx(struct net_device *net_dev, dpa_bp = priv->dpa_bp; WARN_ON(dpa_bp != dpa_bpid2pool(fd->bpid)); - /* prefetch the first 64 bytes of the frame */ + /* prefetch the first 64 bytes of the frame or the SGT start */ dma_unmap_single(dpa_bp->dev, addr, dpa_bp->size, DMA_BIDIRECTIONAL); prefetch(phys_to_virt(addr) + dpa_fd_offset(fd)); - /* The only FD type that we may receive is contig */ - WARN_ON(fd->format != qm_fd_contig); + /* The only FD types that we may receive are contig and S/G */ + WARN_ON((fd->format != qm_fd_contig) && (fd->format != qm_fd_sg)); - skb = contig_fd_to_skb(priv, fd); + if (likely(fd->format == qm_fd_contig)) + skb = contig_fd_to_skb(priv, fd); + else + skb = sg_fd_to_skb(priv, fd, count_ptr); - /* Account for the contig buffer - * having been removed from the pool. + /* Account for either the contig buffer or the SGT buffer (depending on + * which case we were in) having been removed from the pool. */ (*count_ptr)--; skb->protocol = eth_type_trans(skb, net_dev); @@ -334,6 +497,122 @@ static int skb_to_contig_fd(struct dpa_priv *priv, return 0; } +static int skb_to_sg_fd(struct dpa_priv *priv, + struct sk_buff *skb, struct qm_fd *fd) +{ + struct dpa_bp *dpa_bp = priv->dpa_bp; + dma_addr_t addr; + struct sk_buff **skbh; + struct net_device *net_dev = priv->net_dev; + int err; + + struct qm_sg_entry *sgt; + void *sgt_buf; + void *buffer_start; + skb_frag_t *frag; + int i, j; + const enum dma_data_direction dma_dir = DMA_TO_DEVICE; + const int nr_frags = skb_shinfo(skb)->nr_frags; + + fd->format = qm_fd_sg; + + /* get a page frag to store the SGTable */ + sgt_buf = netdev_alloc_frag(priv->tx_headroom + + sizeof(struct qm_sg_entry) * (1 + nr_frags)); + if (unlikely(!sgt_buf)) { + netdev_err(net_dev, "netdev_alloc_frag() failed\n"); + return -ENOMEM; + } + + /* Enable L3/L4 hardware checksum computation. + * + * We must do this before dma_map_single(DMA_TO_DEVICE), because we may + * need to write into the skb. + */ + err = dpa_enable_tx_csum(priv, skb, fd, + sgt_buf + DPA_TX_PRIV_DATA_SIZE); + if (unlikely(err < 0)) { + if (net_ratelimit()) + netif_err(priv, tx_err, net_dev, "HW csum error: %d\n", + err); + goto csum_failed; + } + + sgt = (struct qm_sg_entry *)(sgt_buf + priv->tx_headroom); + sgt[0].bpid = FSL_DPAA_BPID_INV; + sgt[0].offset = 0; + sgt[0].length = cpu_to_be32(skb_headlen(skb)); + sgt[0].extension = 0; + sgt[0].final = 0; + addr = dma_map_single(dpa_bp->dev, skb->data, sgt[0].length, dma_dir); + if (unlikely(dma_mapping_error(dpa_bp->dev, addr))) { + dev_err(dpa_bp->dev, "DMA mapping failed"); + err = -EINVAL; + goto sg0_map_failed; + } + sgt[0].addr_hi = (u8)upper_32_bits(addr); + sgt[0].addr_lo = cpu_to_be32(lower_32_bits(addr)); + + /* populate the rest of SGT entries */ + for (i = 1; i <= nr_frags; i++) { + frag = &skb_shinfo(skb)->frags[i - 1]; + sgt[i].bpid = FSL_DPAA_BPID_INV; + sgt[i].offset = 0; + sgt[i].length = cpu_to_be32(frag->size); + sgt[i].extension = 0; + sgt[i].final = 0; + + WARN_ON(!skb_frag_page(frag)); + addr = skb_frag_dma_map(dpa_bp->dev, frag, 0, sgt[i].length, + dma_dir); + if (unlikely(dma_mapping_error(dpa_bp->dev, addr))) { + dev_err(dpa_bp->dev, "DMA mapping failed"); + err = -EINVAL; + goto sg_map_failed; + } + + /* keep the offset in the address */ + sgt[i].addr_hi = (u8)upper_32_bits(addr); + sgt[i].addr_lo = cpu_to_be32(lower_32_bits(addr)); + } + sgt[i - 1].final = 1; + + fd->length20 = skb->len; + fd->offset = priv->tx_headroom; + + /* DMA map the SGT page */ + buffer_start = (void *)sgt - priv->tx_headroom; + skbh = (struct sk_buff **)buffer_start; + *skbh = skb; + + addr = dma_map_single(dpa_bp->dev, buffer_start, priv->tx_headroom + + sizeof(struct qm_sg_entry) * (1 + nr_frags), + dma_dir); + if (unlikely(dma_mapping_error(dpa_bp->dev, addr))) { + dev_err(dpa_bp->dev, "DMA mapping failed"); + err = -EINVAL; + goto sgt_map_failed; + } + + fd->bpid = FSL_DPAA_BPID_INV; + fd->cmd |= FM_FD_CMD_FCO; + fd->addr_hi = (u8)upper_32_bits(addr); + fd->addr_lo = lower_32_bits(addr); + + return 0; + +sgt_map_failed: +sg_map_failed: + for (j = 0; j < i; j++) + dma_unmap_page(dpa_bp->dev, qm_sg_addr(&sgt[j]), + cpu_to_be32(sgt[j].length), dma_dir); +sg0_map_failed: +csum_failed: + put_page(virt_to_head_page(sgt_buf)); + + return err; +} + int dpa_tx(struct sk_buff *skb, struct net_device *net_dev) { struct dpa_priv *priv; @@ -342,6 +621,7 @@ int dpa_tx(struct sk_buff *skb, struct net_device *net_dev) struct rtnl_link_stats64 *percpu_stats; int err = 0; const int queue_mapping = dpa_get_queue_mapping(skb); + bool nonlinear = skb_is_nonlinear(skb); int *countptr, offset = 0; priv = netdev_priv(net_dev); @@ -351,19 +631,38 @@ int dpa_tx(struct sk_buff *skb, struct net_device *net_dev) clear_fd(&fd); - /* We're going to store the skb backpointer at the beginning - * of the data buffer, so we need a privately owned skb - * - * We've made sure skb is not shared in dev->priv_flags, - * we need to verify the skb head is not cloned - */ - if (skb_cow_head(skb, priv->tx_headroom)) - goto enomem; + if (!nonlinear) { + /* We're going to store the skb backpointer at the beginning + * of the data buffer, so we need a privately owned skb + * + * We've made sure skb is not shared in dev->priv_flags, + * we need to verify the skb head is not cloned + */ + if (skb_cow_head(skb, priv->tx_headroom)) + goto enomem; + + WARN_ON(skb_is_nonlinear(skb)); + } - WARN_ON(skb_is_nonlinear(skb)); + /* MAX_SKB_FRAGS is equal or larger than our DPA_SGT_MAX_ENTRIES; + * make sure we don't feed FMan with more fragments than it supports. + * Btw, we're using the first sgt entry to store the linear part of + * the skb, so we're one extra frag short. + */ + if (nonlinear && + likely(skb_shinfo(skb)->nr_frags < DPA_SGT_MAX_ENTRIES)) { + /* Just create a S/G fd based on the skb */ + err = skb_to_sg_fd(priv, skb, &fd); + } else { + /* If the egress skb contains more fragments than we support + * we have no choice but to linearize it ourselves. + */ + if (unlikely(nonlinear) && __skb_linearize(skb)) + goto enomem; - /* Finally, create a contig FD from this skb */ - err = skb_to_contig_fd(priv, skb, &fd, countptr, &offset); + /* Finally, create a contig FD from this skb */ + err = skb_to_contig_fd(priv, skb, &fd, countptr, &offset); + } if (unlikely(err < 0)) goto skb_to_fd_failed; -- 1.7.11.7 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/