Received: by 2002:ac0:a5a7:0:0:0:0:0 with SMTP id m36-v6csp4086884imm; Sat, 21 Jul 2018 09:48:51 -0700 (PDT) X-Google-Smtp-Source: AAOMgpdzAxzg9vDo5EYCfyHAFNepzUqml+gNNivY4BEa0EDy0LxjyiDT12euRmU/QfGx53PJdzFq X-Received: by 2002:a17:902:e201:: with SMTP id ce1-v6mr6350256plb.136.1532191731301; Sat, 21 Jul 2018 09:48:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532191731; cv=none; d=google.com; s=arc-20160816; b=K9/0Q1uGCJ/7STSYYY/Qtan77iZDaApalxgCVNh5jSeqt1nNuVsWCxx+6fQBXMYcI5 j3Y1P57zci58rrF0U0PuuQYaNGrZ0KhKib7cuUk7nXeIvM7Hor/ewm30FBsPemLxuU+6 DYtHcxSHB/WgsjrxLT9cuNbzR2icIIvWsSceU4REwxh9rommyMNol6F5yMrdMQlF4dwq n5cI+Zup7kPo9flSolDn9i2k2ppdYx0BV301EqfKnfdzXRgm9fQtRYzq0zEA+5gdCniG +B1iP8NxntOUCb54+TGOe40QLdTZyuqq/jU8lDd4L/MGmtkkIb1A+Qfp6DLLssilp34J tlHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:mail-followup-to :message-id:subject:cc:to:from:date:dkim-signature :arc-authentication-results; bh=SKOqPeGd7byZTD94PyPQS9ShDJefWknVGO/mPKn0QeA=; b=IaTyUflvZbllmxDy4HV9G/w86BGLe8Pgns/PpDYHFVAAHD//opxNwNWdDh+B/2uj4E 5foK8nFrGkSp7eM4GglF8GyhFqTgIVmEzFjY1Kd5vOweps50FKyryeA4Yoed3QcR2vA+ OUiM8ScKge12Jt0KtBuAyXfmwrZQcEeW9q3QQXh1JkeG69p2mFQYn8j1dQAv9A52Wvv6 YnjlhPof1GAy9M9+gs68oiUJTR7zezEzGfVsRROceW9r0zCpHvm7Hjvy5JuT63EMsI3W GmWvnwnzRTrycS3N6Mr6S6YnE7ls9gXqVOR9bYbtl3S8cUjpJlr2yBwnUx8gN1dhu2ie cUoQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=W4hL0IE2; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 3-v6si4065944plc.415.2018.07.21.09.48.21; Sat, 21 Jul 2018 09:48:51 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=W4hL0IE2; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728185AbeGURkQ (ORCPT + 99 others); Sat, 21 Jul 2018 13:40:16 -0400 Received: from mail-lj1-f193.google.com ([209.85.208.193]:32993 "EHLO mail-lj1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728144AbeGURkP (ORCPT ); Sat, 21 Jul 2018 13:40:15 -0400 Received: by mail-lj1-f193.google.com with SMTP id s12-v6so13566568ljj.0 for ; Sat, 21 Jul 2018 09:46:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-disposition:in-reply-to:user-agent; bh=SKOqPeGd7byZTD94PyPQS9ShDJefWknVGO/mPKn0QeA=; b=W4hL0IE2w+y1GHSYkCxZnoBCnRgrioviy/Mjk//D2w1Z70Bdp5QZmKj5pPZHW//N3U RrpI9AEaujOkkF3kEZRFAJ6sEZjNZt+FSs+NRdx2FqMEp0qG+dDcvp0YUAd80PgNKQ+l TSDP5cpPk6EQf9aKzgz2yTiE1wSZxduW5RB58= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :in-reply-to:user-agent; bh=SKOqPeGd7byZTD94PyPQS9ShDJefWknVGO/mPKn0QeA=; b=L0mumCvf2mXlJbyOdWcoAGhx+4KdGzyvmarI+x/otVHH84Oue17FZ5xdNgjrNw5oMc y7M5eGRhwGp9fF5OSNfG0Vf5sCqy1a9ZpxLs/ltIE0+DcTLYbwSf6nlNFTiCemT7VYqK EOjN5c44gCyiHd18BrICeftVTjB0grnF5FVj0AcXnv4aRbo0p3Cg4Y3KV6NUsQHJGQxm t8THvyvbE2sEgS2tuKEiRa7Z2tMUfO5CeuW/VfCMaAZ3Nu5IJTWS0aeXu3i+I0ncXMHO 68qBjPMk+Y9Vm6U3AmdLJVaqjC1B49OGHjD9FLRmoTuoydLTwDKmJX52gAKMtaTOR0kD YZsA== X-Gm-Message-State: AOUpUlEA71JDCoFaMEgtvjGBbQ+jkPlgQCETpeCE7duevPe7Fu9N6DlU g34rPUonfXO3iFCrXiSv4feK5A== X-Received: by 2002:a2e:9c0f:: with SMTP id s15-v6mr4796975lji.97.1532191612160; Sat, 21 Jul 2018 09:46:52 -0700 (PDT) Received: from khorivan (59-201-94-178.pool.ukrtel.net. [178.94.201.59]) by smtp.gmail.com with ESMTPSA id f129-v6sm725388lff.37.2018.07.21.09.46.50 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 21 Jul 2018 09:46:51 -0700 (PDT) Date: Sat, 21 Jul 2018 19:46:49 +0300 From: Ivan Khoronzhuk To: Richard Cochran Cc: davem@davemloft.net, grygorii.strashko@ti.com, corbet@lwn.net, akpm@linux-foundation.org, netdev@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-omap@vger.kernel.org, vinicius.gomes@intel.com, henrik@austad.us, jesus.sanchez-palencia@intel.com, ilias.apalodimas@linaro.org, p-varis@ti.com, spatton@ti.com, francois.ozog@linaro.org, yogeshs@ti.com, nsekhar@ti.com, andrew@lunn.ch Subject: Re: [PATCH v4 net-next 6/6] Documentation: networking: cpsw: add MQPRIO & CBS offload examples Message-ID: <20180721164643.GB2209@khorivan> Mail-Followup-To: Richard Cochran , davem@davemloft.net, grygorii.strashko@ti.com, corbet@lwn.net, akpm@linux-foundation.org, netdev@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-omap@vger.kernel.org, vinicius.gomes@intel.com, henrik@austad.us, jesus.sanchez-palencia@intel.com, ilias.apalodimas@linaro.org, p-varis@ti.com, spatton@ti.com, francois.ozog@linaro.org, yogeshs@ti.com, nsekhar@ti.com, andrew@lunn.ch References: <20180721115923.1389-1-ivan.khoronzhuk@linaro.org> <20180721115923.1389-7-ivan.khoronzhuk@linaro.org> <20180721151028.fxjlxordxpsiohwo@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <20180721151028.fxjlxordxpsiohwo@localhost> User-Agent: Mutt/1.9.4 (2018-02-28) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, Jul 21, 2018 at 08:10:29AM -0700, Richard Cochran wrote: >On Sat, Jul 21, 2018 at 02:59:23PM +0300, Ivan Khoronzhuk wrote: >> This document describes MQPRIO and CBS Qdisc offload configuration >> for cpsw driver based on examples. It potentially can be used in >> audio video bridging (AVB) and time sensitive networking (TSN). >> >> Reviewed-by: Grygorii Strashko >> Signed-off-by: Ivan Khoronzhuk >> --- >> Documentation/networking/ti-cpsw.txt | 540 +++++++++++++++++++++++++++ >> 1 file changed, 540 insertions(+) >> create mode 100644 Documentation/networking/ti-cpsw.txt >> >> diff --git a/Documentation/networking/ti-cpsw.txt b/Documentation/networking/ti-cpsw.txt >> new file mode 100644 >> index 000000000000..1e840346cae1 >> --- /dev/null >> +++ b/Documentation/networking/ti-cpsw.txt >> @@ -0,0 +1,540 @@ >> +* Texas Instruments CPSW ethernet driver >> + >> +Multiqueue & CBS & MQPRIO >> +===================================================================== >> +===================================================================== >> + >> +The cpsw has 3 CBS shapers for each external ports. This document >> +describes MQPRIO and CBS Qdisc offload configuration for cpsw driver >> +based on examples. It potentially can be used in audio video bridging >> +(AVB) and time sensitive networking (TSN). >> + >> +The following examples was tested on AM572x EVM and BBB boards. >> + >> +Test setup >> +========== >> + >> +Under consideration two examples with AM52xx EVM running cpsw driver >> +in dual_emac mode. > >s/AM52xx/AM572x ? yes, AM572x > >Also, there are more mentions of AM52xx, below. I'll correct. > >Thanks, >Richard > > >> + >> +Several prerequisites: >> +- TX queues must be rated starting from txq0 that has highest priority >> +- Traffic classes are used starting from 0, that has highest priority >> +- CBS shapers should be used with rated queues >> +- The bandwidth for CBS shapers has to be set a little bit more then >> + potential incoming rate, thus, rate of all incoming tx queues has >> + to be a little less >> +- Real rates can differ, due to discreetness >> +- Map skb-priority to txq is not enough, also skb-priority to l2 prio >> + map has to be created with ip or vconfig tool >> +- Any l2/socket prio (0 - 7) for classes can be used, but for >> + simplicity default values are used: 3 and 2 >> +- only 2 classes tested: A and B, but checked and can work with more, >> + maximum allowed 4, but only for 3 rate can be set. >> + >> +Test setup for examples >> +======================= >> + +-------------------------------+ >> + |--+ | >> + | | Workstation0 | >> + |E | MAC 18:03:73:66:87:42 | >> ++-----------------------------+ +--|t | | >> +| | 1 | E | | |h |./tsn_listener -d \ | >> +| Target board: | 0 | t |--+ |0 | 18:03:73:66:87:42 -i eth0 \| >> +| AM572x EVM | 0 | h | | | -s 1500 | >> +| | 0 | 0 | |--+ | >> +| Only 2 classes: |Mb +---| +-------------------------------+ >> +| class A, class B | | >> +| | +---| +-------------------------------+ >> +| | 1 | E | |--+ | >> +| | 0 | t | | | Workstation1 | >> +| | 0 | h |--+ |E | MAC 20:cf:30:85:7d:fd | >> +| |Mb | 1 | +--|t | | >> ++-----------------------------+ |h |./tsn_listener -d \ | >> + |0 | 20:cf:30:85:7d:fd -i eth0 \| >> + | | -s 1500 | >> + |--+ | >> + +-------------------------------+ >> + >> +********************************************************************* >> +********************************************************************* >> +********************************************************************* >> +Example 1: One port tx AVB configuration scheme for target board >> +---------------------------------------------------------------------- >> +(prints and scheme for AM52xx evm, applicable for single port boards) >> + >> +tc - traffic class >> +txq - transmit queue >> +p - priority >> +f - fifo (cpsw fifo) >> +S - shaper configured >> + >> ++------------------------------------------------------------------+ u >> +| +---------------+ +---------------+ +------+ +------+ | s >> +| | | | | | | | | | e >> +| | App 1 | | App 2 | | Apps | | Apps | | r >> +| | Class A | | Class B | | Rest | | Rest | | >> +| | Eth0 | | Eth0 | | Eth0 | | Eth1 | | s >> +| | VLAN100 | | VLAN100 | | | | | | | | p >> +| | 40 Mb/s | | 20 Mb/s | | | | | | | | a >> +| | SO_PRIORITY=3 | | SO_PRIORITY=2 | | | | | | | | c >> +| | | | | | | | | | | | | | e >> +| +---|-----------+ +---|-----------+ +---|--+ +---|--+ | >> ++-----|------------------|------------------|--------|-------------+ >> + +-+ +------------+ | | >> + | | +-----------------+ +--+ >> + | | | | >> ++---|-------|-------------|-----------------------|----------------+ >> +| +----+ +----+ +----+ +----+ +----+ | >> +| | p3 | | p2 | | p1 | | p0 | | p0 | | k >> +| \ / \ / \ / \ / \ / | e >> +| \ / \ / \ / \ / \ / | r >> +| \/ \/ \/ \/ \/ | n >> +| | | | | | e >> +| | | +-----+ | | l >> +| | | | | | >> +| +----+ +----+ +----+ +----+ | s >> +| |tc0 | |tc1 | |tc2 | |tc0 | | p >> +| \ / \ / \ / \ / | a >> +| \ / \ / \ / \ / | c >> +| \/ \/ \/ \/ | e >> +| | | +-----+ | | >> +| | | | | | | >> +| | | | | | | >> +| | | | | | | >> +| +----+ +----+ +----+ +----+ +----+ | >> +| |txq0| |txq1| |txq2| |txq3| |txq4| | >> +| \ / \ / \ / \ / \ / | >> +| \ / \ / \ / \ / \ / | >> +| \/ \/ \/ \/ \/ | >> +| +-|------|------|------|--+ +--|--------------+ | >> +| | | | | | | Eth0.100 | | Eth1 | | >> ++---|------|------|------|------------------------|----------------+ >> + | | | | | >> + p p p p | >> + 3 2 0-1, 4-7 <- L2 priority | >> + | | | | | >> + | | | | | >> ++---|------|------|------|------------------------|----------------+ >> +| | | | | |----------+ | >> +| +----+ +----+ +----+ +----+ +----+ | >> +| |dma7| |dma6| |dma5| |dma4| |dma3| | >> +| \ / \ / \ / \ / \ / | c >> +| \S / \S / \ / \ / \ / | p >> +| \/ \/ \/ \/ \/ | s >> +| | | | +----- | | w >> +| | | | | | | >> +| | | | | | | d >> +| +----+ +----+ +----+p p+----+ | r >> +| | | | | | |o o| | | i >> +| | f3 | | f2 | | f0 |r r| f0 | | v >> +| |tc0 | |tc1 | |tc2 |t t|tc0 | | e >> +| \CBS / \CBS / \CBS /1 2\CBS / | r >> +| \S / \S / \ / \ / | >> +| \/ \/ \/ \/ | >> ++------------------------------------------------------------------+ >> +========================================Eth==========================> >> + >> +1) >> +// Add 4 tx queues, for interface Eth0, and 1 tx queue for Eth1 >> +$ ethtool -L eth0 rx 1 tx 5 >> +rx unmodified, ignoring >> + >> +2) >> +// Check if num of queues is set correctly: >> +$ ethtool -l eth0 >> +Channel parameters for eth0: >> +Pre-set maximums: >> +RX: 8 >> +TX: 8 >> +Other: 0 >> +Combined: 0 >> +Current hardware settings: >> +RX: 1 >> +TX: 5 >> +Other: 0 >> +Combined: 0 >> + >> +3) >> +// TX queues must be rated starting from 0, so set bws for tx0 and tx1 >> +// Set rates 40 and 20 Mb/s appropriately. >> +// Pay attention, real speed can differ a bit due to discreetness. >> +// Leave last 2 tx queues not rated. >> +$ echo 40 > /sys/class/net/eth0/queues/tx-0/tx_maxrate >> +$ echo 20 > /sys/class/net/eth0/queues/tx-1/tx_maxrate >> + >> +4) >> +// Check maximum rate of tx (cpdma) queues: >> +$ cat /sys/class/net/eth0/queues/tx-*/tx_maxrate >> +40 >> +20 >> +0 >> +0 >> +0 >> + >> +5) >> +// Map skb->priority to traffic class: >> +// 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2 >> +// Map traffic class to transmit queue: >> +// tc0 -> txq0, tc1 -> txq1, tc2 -> (txq2, txq3) >> +$ tc qdisc replace dev eth0 handle 100: parent root mqprio num_tc 3 \ >> +map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@0 1@1 2@2 hw 1 >> + >> +5a) >> +// As two interface sharing same set of tx queues, assign all traffic >> +// coming to interface Eth1 to separate queue in order to not mix it >> +// with traffic from interface Eth0, so use separate txq to send >> +// packets to Eth1, so all prio -> tc0 and tc0 -> txq4 >> +// Here hw 0, so here still default configuration for eth1 in hw >> +$ tc qdisc replace dev eth1 handle 100: parent root mqprio num_tc 1 \ >> +map 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 queues 1@4 hw 0 >> + >> +6) >> +// Check classes settings >> +$ tc -g class show dev eth0 >> ++---(100:ffe2) mqprio >> +| +---(100:3) mqprio >> +| +---(100:4) mqprio >> +| >> ++---(100:ffe1) mqprio >> +| +---(100:2) mqprio >> +| >> ++---(100:ffe0) mqprio >> + +---(100:1) mqprio >> + >> +$ tc -g class show dev eth1 >> ++---(100:ffe0) mqprio >> + +---(100:5) mqprio >> + >> +7) >> +// Set rate for class A - 41 Mbit (tc0, txq0) using CBS Qdisc >> +// Set it +1 Mb for reserve (important!) >> +// here only idle slope is important, others arg are ignored >> +// Pay attention, real speed can differ a bit due to discreetness >> +$ tc qdisc add dev eth0 parent 100:1 cbs locredit -1438 \ >> +hicredit 62 sendslope -959000 idleslope 41000 offload 1 >> +net eth0: set FIFO3 bw = 50 >> + >> +8) >> +// Set rate for class B - 21 Mbit (tc1, txq1) using CBS Qdisc: >> +// Set it +1 Mb for reserve (important!) >> +$ tc qdisc add dev eth0 parent 100:2 cbs locredit -1468 \ >> +hicredit 65 sendslope -979000 idleslope 21000 offload 1 >> +net eth0: set FIFO2 bw = 30 >> + >> +9) >> +// Create vlan 100 to map sk->priority to vlan qos >> +$ ip link add link eth0 name eth0.100 type vlan id 100 >> +8021q: 802.1Q VLAN Support v1.8 >> +8021q: adding VLAN 0 to HW filter on device eth0 >> +8021q: adding VLAN 0 to HW filter on device eth1 >> +net eth0: Adding vlanid 100 to vlan filter >> + >> +10) >> +// Map skb->priority to L2 prio, 1 to 1 >> +$ ip link set eth0.100 type vlan \ >> +egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 >> + >> +11) >> +// Check egress map for vlan 100 >> +$ cat /proc/net/vlan/eth0.100 >> +[...] >> +INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0 >> +EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 >> + >> +12) >> +// Run your appropriate tools with socket option "SO_PRIORITY" >> +// to 3 for class A and/or to 2 for class B >> +// (I took at https://www.spinics.net/lists/netdev/msg460869.html) >> +./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p3 -s 1500& >> +./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p2 -s 1500& >> + >> +13) >> +// run your listener on workstation (should be in same vlan) >> +// (I took at https://www.spinics.net/lists/netdev/msg460869.html) >> +./tsn_listener -d 18:03:73:66:87:42 -i enp5s0 -s 1500 >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39000 kbps >> + >> +14) >> +// Restore default configuration if needed >> +$ ip link del eth0.100 >> +$ tc qdisc del dev eth1 root >> +$ tc qdisc del dev eth0 root >> +net eth0: Prev FIFO2 is shaped >> +net eth0: set FIFO3 bw = 0 >> +net eth0: set FIFO2 bw = 0 >> +$ ethtool -L eth0 rx 1 tx 1 >> + >> +********************************************************************* >> +********************************************************************* >> +********************************************************************* >> +Example 2: Two port tx AVB configuration scheme for target board >> +---------------------------------------------------------------------- >> +(prints and scheme for AM52xx evm, for dual emac boards only) >> + >> ++------------------------------------------------------------------+ u >> +| +----------+ +----------+ +------+ +----------+ +----------+ | s >> +| | | | | | | | | | | | e >> +| | App 1 | | App 2 | | Apps | | App 3 | | App 4 | | r >> +| | Class A | | Class B | | Rest | | Class B | | Class A | | >> +| | Eth0 | | Eth0 | | | | | Eth1 | | Eth1 | | s >> +| | VLAN100 | | VLAN100 | | | | | VLAN100 | | VLAN100 | | p >> +| | 40 Mb/s | | 20 Mb/s | | | | | 10 Mb/s | | 30 Mb/s | | a >> +| | SO_PRI=3 | | SO_PRI=2 | | | | | SO_PRI=3 | | SO_PRI=2 | | c >> +| | | | | | | | | | | | | | | | | e >> +| +---|------+ +---|------+ +---|--+ +---|------+ +---|------+ | >> ++-----|-------------|-------------|---------|-------------|--------+ >> + +-+ +-------+ | +----------+ +----+ >> + | | +-------+------+ | | >> + | | | | | | >> ++---|-------|-------------|--------------|-------------|-------|---+ >> +| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | >> +| | p3 | | p2 | | p1 | | p0 | | p0 | | p1 | | p2 | | p3 | | k >> +| \ / \ / \ / \ / \ / \ / \ / \ / | e >> +| \ / \ / \ / \ / \ / \ / \ / \ / | r >> +| \/ \/ \/ \/ \/ \/ \/ \/ | n >> +| | | | | | | | e >> +| | | +----+ +----+ | | | l >> +| | | | | | | | >> +| +----+ +----+ +----+ +----+ +----+ +----+ | s >> +| |tc0 | |tc1 | |tc2 | |tc2 | |tc1 | |tc0 | | p >> +| \ / \ / \ / \ / \ / \ / | a >> +| \ / \ / \ / \ / \ / \ / | c >> +| \/ \/ \/ \/ \/ \/ | e >> +| | | +-----+ +-----+ | | | >> +| | | | | | | | | | >> +| | | | | | | | | | >> +| | | | | E E | | | | | >> +| +----+ +----+ +----+ +----+ t t +----+ +----+ +----+ +----+ | >> +| |txq0| |txq1| |txq4| |txq5| h h |txq6| |txq7| |txq3| |txq2| | >> +| \ / \ / \ / \ / 0 1 \ / \ / \ / \ / | >> +| \ / \ / \ / \ / . . \ / \ / \ / \ / | >> +| \/ \/ \/ \/ 1 1 \/ \/ \/ \/ | >> +| +-|------|------|------|--+ 0 0 +-|------|------|------|--+ | >> +| | | | | | | 0 0 | | | | | | | >> ++---|------|------|------|---------------|------|------|------|----+ >> + | | | | | | | | >> + p p p p p p p p >> + 3 2 0-1, 4-7 <-L2 pri-> 0-1, 4-7 2 3 >> + | | | | | | | | >> + | | | | | | | | >> ++---|------|------|------|---------------|------|------|------|----+ >> +| | | | | | | | | | >> +| +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ | >> +| |dma7| |dma6| |dma3| |dma2| |dma1| |dma0| |dma4| |dma5| | >> +| \ / \ / \ / \ / \ / \ / \ / \ / | c >> +| \S / \S / \ / \ / \ / \ / \S / \S / | p >> +| \/ \/ \/ \/ \/ \/ \/ \/ | s >> +| | | | +----- | | | | | w >> +| | | | | +----+ | | | | >> +| | | | | | | | | | d >> +| +----+ +----+ +----+p p+----+ +----+ +----+ | r >> +| | | | | | |o o| | | | | | | i >> +| | f3 | | f2 | | f0 |r CPSW r| f3 | | f2 | | f0 | | v >> +| |tc0 | |tc1 | |tc2 |t t|tc0 | |tc1 | |tc2 | | e >> +| \CBS / \CBS / \CBS /1 2\CBS / \CBS / \CBS / | r >> +| \S / \S / \ / \S / \S / \ / | >> +| \/ \/ \/ \/ \/ \/ | >> ++------------------------------------------------------------------+ >> +========================================Eth==========================> >> + >> +1) >> +// Add 8 tx queues, for interface Eth0, but they are common, so are accessed >> +// by two interfaces Eth0 and Eth1. >> +$ ethtool -L eth1 rx 1 tx 8 >> +rx unmodified, ignoring >> + >> +2) >> +// Check if num of queues is set correctly: >> +$ ethtool -l eth0 >> +Channel parameters for eth0: >> +Pre-set maximums: >> +RX: 8 >> +TX: 8 >> +Other: 0 >> +Combined: 0 >> +Current hardware settings: >> +RX: 1 >> +TX: 8 >> +Other: 0 >> +Combined: 0 >> + >> +3) >> +// TX queues must be rated starting from 0, so set bws for tx0 and tx1 for Eth0 >> +// and for tx2 and tx3 for Eth1. That is, rates 40 and 20 Mb/s appropriately >> +// for Eth0 and 30 and 10 Mb/s for Eth1. >> +// Real speed can differ a bit due to discreetness >> +// Leave last 4 tx queues as not rated >> +$ echo 40 > /sys/class/net/eth0/queues/tx-0/tx_maxrate >> +$ echo 20 > /sys/class/net/eth0/queues/tx-1/tx_maxrate >> +$ echo 30 > /sys/class/net/eth1/queues/tx-2/tx_maxrate >> +$ echo 10 > /sys/class/net/eth1/queues/tx-3/tx_maxrate >> + >> +4) >> +// Check maximum rate of tx (cpdma) queues: >> +$ cat /sys/class/net/eth0/queues/tx-*/tx_maxrate >> +40 >> +20 >> +30 >> +10 >> +0 >> +0 >> +0 >> +0 >> + >> +5) >> +// Map skb->priority to traffic class for Eth0: >> +// 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2 >> +// Map traffic class to transmit queue: >> +// tc0 -> txq0, tc1 -> txq1, tc2 -> (txq4, txq5) >> +$ tc qdisc replace dev eth0 handle 100: parent root mqprio num_tc 3 \ >> +map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@0 1@1 2@4 hw 1 >> + >> +6) >> +// Check classes settings >> +$ tc -g class show dev eth0 >> ++---(100:ffe2) mqprio >> +| +---(100:5) mqprio >> +| +---(100:6) mqprio >> +| >> ++---(100:ffe1) mqprio >> +| +---(100:2) mqprio >> +| >> ++---(100:ffe0) mqprio >> + +---(100:1) mqprio >> + >> +7) >> +// Set rate for class A - 41 Mbit (tc0, txq0) using CBS Qdisc for Eth0 >> +// here only idle slope is important, others ignored >> +// Real speed can differ a bit due to discreetness >> +$ tc qdisc add dev eth0 parent 100:1 cbs locredit -1470 \ >> +hicredit 62 sendslope -959000 idleslope 41000 offload 1 >> +net eth0: set FIFO3 bw = 50 >> + >> +8) >> +// Set rate for class B - 21 Mbit (tc1, txq1) using CBS Qdisc for Eth0 >> +$ tc qdisc add dev eth0 parent 100:2 cbs locredit -1470 \ >> +hicredit 65 sendslope -979000 idleslope 21000 offload 1 >> +net eth0: set FIFO2 bw = 30 >> + >> +9) >> +// Create vlan 100 to map sk->priority to vlan qos for Eth0 >> +$ ip link add link eth0 name eth0.100 type vlan id 100 >> +net eth0: Adding vlanid 100 to vlan filter >> + >> +10) >> +// Map skb->priority to L2 prio for Eth0.100, one to one >> +$ ip link set eth0.100 type vlan \ >> +egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 >> + >> +11) >> +// Check egress map for vlan 100 >> +$ cat /proc/net/vlan/eth0.100 >> +[...] >> +INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0 >> +EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 >> + >> +12) >> +// Map skb->priority to traffic class for Eth1: >> +// 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2 >> +// Map traffic class to transmit queue: >> +// tc0 -> txq2, tc1 -> txq3, tc2 -> (txq6, txq7) >> +$ tc qdisc replace dev eth1 handle 100: parent root mqprio num_tc 3 \ >> +map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@2 1@3 2@6 hw 1 >> + >> +13) >> +// Check classes settings >> +$ tc -g class show dev eth1 >> ++---(100:ffe2) mqprio >> +| +---(100:7) mqprio >> +| +---(100:8) mqprio >> +| >> ++---(100:ffe1) mqprio >> +| +---(100:4) mqprio >> +| >> ++---(100:ffe0) mqprio >> + +---(100:3) mqprio >> + >> +14) >> +// Set rate for class A - 31 Mbit (tc0, txq2) using CBS Qdisc for Eth1 >> +// here only idle slope is important, others ignored >> +// Set it +1 Mb for reserve (important!) >> +$ tc qdisc add dev eth1 parent 100:3 cbs locredit -1453 \ >> +hicredit 47 sendslope -969000 idleslope 31000 offload 1 >> +net eth1: set FIFO3 bw = 31 >> + >> +15) >> +// Set rate for class B - 11 Mbit (tc1, txq3) using CBS Qdisc for Eth1 >> +// Set it +1 Mb for reserve (important!) >> +$ tc qdisc add dev eth1 parent 100:4 cbs locredit -1483 \ >> +hicredit 34 sendslope -989000 idleslope 11000 offload 1 >> +net eth1: set FIFO2 bw = 11 >> + >> +16) >> +// Create vlan 100 to map sk->priority to vlan qos for Eth1 >> +$ ip link add link eth1 name eth1.100 type vlan id 100 >> +net eth1: Adding vlanid 100 to vlan filter >> + >> +17) >> +// Map skb->priority to L2 prio for Eth1.100, one to one >> +$ ip link set eth1.100 type vlan \ >> +egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 >> + >> +18) >> +// Check egress map for vlan 100 >> +$ cat /proc/net/vlan/eth1.100 >> +[...] >> +INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0 >> +EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 >> + >> +19) >> +// Run appropriate tools with socket option "SO_PRIORITY" to 3 >> +// for class A and to 2 for class B. For both interfaces >> +./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p2 -s 1500& >> +./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p3 -s 1500& >> +./tsn_talker -d 20:cf:30:85:7d:fd -i eth1.100 -p2 -s 1500& >> +./tsn_talker -d 20:cf:30:85:7d:fd -i eth1.100 -p3 -s 1500& >> + >> +20) >> +// run your listener on workstation (should be in same vlan) >> +// (I took at https://www.spinics.net/lists/netdev/msg460869.html) >> +./tsn_listener -d 18:03:73:66:87:42 -i enp5s0 -s 1500 >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39012 kbps >> +Receiving data rate: 39000 kbps >> + >> +21) >> +// Restore default configuration if needed >> +$ ip link del eth1.100 >> +$ ip link del eth0.100 >> +$ tc qdisc del dev eth1 root >> +net eth1: Prev FIFO2 is shaped >> +net eth1: set FIFO3 bw = 0 >> +net eth1: set FIFO2 bw = 0 >> +$ tc qdisc del dev eth0 root >> +net eth0: Prev FIFO2 is shaped >> +net eth0: set FIFO3 bw = 0 >> +net eth0: set FIFO2 bw = 0 >> +$ ethtool -L eth0 rx 1 tx 1 >> -- >> 2.17.1 >> -- Regards, Ivan Khoronzhuk