Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Vladimir Oltean <vladimir.oltean@nxp.com>
To:     netdev@vger.kernel.org
Cc:     linux-kernel@vger.kernel.org,
        "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Florian Fainelli <f.fainelli@gmail.com>,
        Vivien Didelot <vivien.didelot@gmail.com>,
        Andrew Lunn <andrew@lunn.ch>,
        Vladimir Oltean <olteanv@gmail.com>,
        Claudiu Manoil <claudiu.manoil@nxp.com>,
        Alexandre Belloni <alexandre.belloni@bootlin.com>,
        UNGLinuxDriver@microchip.com,
        Colin Foster <colin.foster@in-advantage.com>,
        Roopa Prabhu <roopa@nvidia.com>,
        Nikolay Aleksandrov <razor@blackwall.org>,
        Tobias Waldekranz <tobias@waldekranz.com>,
        =?UTF-8?q?Marek=20Beh=C3=BAn?= <kabel@kernel.org>,
        Ansuel Smith <ansuelsmth@gmail.com>,
        DENG Qingfang <dqfext@gmail.com>,
        =?UTF-8?q?Alvin=20=C5=A0ipraga?= <alsi@bang-olufsen.dk>,
        Linus Walleij <linus.walleij@linaro.org>,
        Luiz Angelo Daros de Luca <luizluca@gmail.com>,
        Felix Fietkau <nbd@nbd.name>, John Crispin <john@phrozen.org>,
        Sean Wang <sean.wang@mediatek.com>,
        David Ahern <dsahern@kernel.org>,
        Stephen Hemminger <stephen@networkplumber.org>
Subject: [PATCH v2 net-next 09/10] docs: net: dsa: update information about multiple CPU ports
Date:   Sun, 11 Sep 2022 04:07:05 +0300
Message-Id: <20220911010706.2137967-10-vladimir.oltean@nxp.com>
In-Reply-To: <20220911010706.2137967-1-vladimir.oltean@nxp.com>
References: <20220911010706.2137967-1-vladimir.oltean@nxp.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Cx3HMqFFGOZeWMV9H3YWbBRMfNMOyxUTkuiMRKqhPmAnkX+WcnjOx0l4A1mY?=
 =?us-ascii?Q?GIM24aMdOTXyXYRFY8YItCfgtIXGtOb+0WxYIzE1EfZbmbXdGBkuuKQ7gM59?=
 =?us-ascii?Q?1zQwl2sxTB2ZX/picQLqLmAcnioqjvOcEQhnSpdZgjSdvPHFQ7i+x4dLNSxi?=
 =?us-ascii?Q?xe3Vh3eWJg09YgJg4bGkStTOd3ecmKsBn4i0yPpRFI57fZKD+jTvrMv6W+lk?=
 =?us-ascii?Q?aaQhx8cWo48XdbG+AngMv4qPviYQH1KL+hmOG1uYpHkTct9HBsZ+rIytYN0U?=
 =?us-ascii?Q?OAZ7CSqgrb7Nf0+iSi/4DV/XGaUW7GhKkdpAE7iFE7GuRUbGYq+Z8trOqq8g?=
 =?us-ascii?Q?dQFhZs8GcW896ek4ARcOZnbqp8XXYG92DK1iGzSAH8OethkFxHYpOhwckbcG?=
 =?us-ascii?Q?ixE0x703L6l+Aoga3w2nZph8KhXh8MBtVi3nDqKTEJF9tM7kgUAzZ9A+F74h?=
 =?us-ascii?Q?ijf3+jHVxjCVjZ4/HzSEj5vfdh7TJsnrn9dVgvQI5VvMsL69yRfwmqNGp7c7?=
 =?us-ascii?Q?xdVdL7/R/JqIfoYrtDhoFXkQTKLomseYUOOahTXzguh5Sj7IfqVeQSY3ZCsS?=
 =?us-ascii?Q?URjLo3foB1ml8XfvENFpvv/0pkqsrKtS18NtwmhC87bcRnFMAim4qYOv/jzV?=
 =?us-ascii?Q?xsdYOjkdoCpzL0bqWf0BZX4vqESbk3zzB4lkaCX9zh3w2wLMKJB3tSWRZsCf?=
 =?us-ascii?Q?tk/FizojNDl9+WUazRFPexPCp7+tR/eFks6jFjXEc7SU3ybutr2qK1grC7z3?=
 =?us-ascii?Q?GrwrINldPW4hbVYRzABvl3plEMcM+3jS/qWu2IU75a5AknWk2feSRUN24tFt?=
 =?us-ascii?Q?iTKMJoHNDYPXirUxJBQ0hfqjFXlD5IzUKpQ3Sp24aHJMvqAerWGqqv7dAfhI?=
 =?us-ascii?Q?N8TO5gg1VQVyqNoXqMaxMmEIHLE5jVjswX7FRYNHdBcGFPY3UOT40zI7ryAA?=
 =?us-ascii?Q?sTx1m7HQrCBtHUm/O8fjc/J5tXaD8OYBwo7j7RsulMiu3BR0UFCTsw+kbqd+?=
 =?us-ascii?Q?xEEIud3Ga9BSt1hiFSHwrdB0oXFMYZDVeHopRYyHubTg9t0qYUEu1yMFEbW8?=
 =?us-ascii?Q?/3sXI5WFtO0iOh42dVPBuFWd9/wWak2dpJgZr6tukS2Z7alVXm3VwBREOp/X?=
 =?us-ascii?Q?Pvg6GtJxDKDiYaSRdhwW8NhMt0NxeUIfKw/zIUbFgMowdu4js4KZj016yOr5?=
 =?us-ascii?Q?Mv57lugpT3LJpZc/aRdMo7QC1fD1/K34MGWM7LhzPbk3UF5RuSdBjMPD+Koi?=
 =?us-ascii?Q?1dF+rwTRA8JjcSECLlOf7Jj3eSiVKs/4RXK8gwiJ6sVNAmMzja9HbdqQUV5v?=
 =?us-ascii?Q?fhuhWWNUdSpdWmND2vBn0xVDMehZBspY9B6IppIiF8im/WR7ke8HLRp0aiy9?=
 =?us-ascii?Q?VhmikMDp6H2PtZXIc42xLIo0MR2P9hBrKCp56v0YnOrddwBtYDCKRdaH1V7Z?=
 =?us-ascii?Q?Lx3k0BKxpqvAc+BW6wkyEjOrOL50ds0k8U9tcC7BFE+4roGD8VKgk1OP2xW+?=
 =?us-ascii?Q?2d2j/1OgpThDi+wH4HNp/WNapKbidicoTjVSfufNUda+9iuxcn/Xt7fwKexD?=
 =?us-ascii?Q?EOjYwTFgYrWtMV3wAOinA+CVSZdd5bQiAGP+eSrwZw8o1UpjRQp4HgSCQVZt?=
 =?us-ascii?Q?uQ=3D=3D?=
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 516e7ce0-1e92-41d9-6c6e-08da9392145e
X-MS-Exchange-CrossTenant-AuthSource: AM0PR04MB5121.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Sep 2022 01:08:04.3837
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ugS/BLk47vC/WtLLWmFeE6VgAiqH6HulWY1Mkot1HABmfxzARnHAvyiB5g1rssGapy3d6qwtJAvWGoAQy31/gw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR04MB7739
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

DSA now supports multiple CPU ports, explain the use cases that are
covered, the new UAPI, the permitted degrees of freedom, the driver API,
and remove some old "hanging fruits".

Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
v1->v2: provide alternate description of how to put CPU ports in a LAG

 .../networking/dsa/configuration.rst          | 96 +++++++++++++++++++
 Documentation/networking/dsa/dsa.rst          | 38 ++++++--
 2 files changed, 128 insertions(+), 6 deletions(-)

diff --git a/Documentation/networking/dsa/configuration.rst b/Documentation/networking/dsa/configuration.rst
index 2b08f1a772d3..827701f8cbfe 100644
--- a/Documentation/networking/dsa/configuration.rst
+++ b/Documentation/networking/dsa/configuration.rst
@@ -49,6 +49,9 @@ In this documentation the following Ethernet interfaces are used:
 *eth0*
   the master interface
 
+*eth1*
+  another master interface
+
 *lan1*
   a slave interface
 
@@ -360,3 +363,96 @@ the ``self`` flag) has been removed. This results in the following changes:
 
 Script writers are therefore encouraged to use the ``master static`` set of
 flags when working with bridge FDB entries on DSA switch interfaces.
+
+Affinity of user ports to CPU ports
+-----------------------------------
+
+Typically, DSA switches are attached to the host via a single Ethernet
+interface, but in cases where the switch chip is discrete, the hardware design
+may permit the use of 2 or more ports connected to the host, for an increase in
+termination throughput.
+
+DSA can make use of multiple CPU ports in two ways. First, it is possible to
+statically assign the termination traffic associated with a certain user port
+to be processed by a certain CPU port. This way, user space can implement
+custom policies of static load balancing between user ports, by spreading the
+affinities according to the available CPU ports.
+
+Secondly, it is possible to perform load balancing between CPU ports on a per
+packet basis, rather than statically assigning user ports to CPU ports.
+This can be achieved by placing the DSA masters under a LAG interface (bonding
+or team). DSA monitors this operation and creates a mirror of this software LAG
+on the CPU ports facing the physical DSA masters that constitute the LAG slave
+devices.
+
+To make use of multiple CPU ports, the firmware (device tree) description of
+the switch must mark all the links between CPU ports and their DSA masters
+using the ``ethernet`` reference/phandle. At startup, only a single CPU port
+and DSA master will be used - the numerically first port from the firmware
+description which has an ``ethernet`` property. It is up to the user to
+configure the system for the switch to use other masters.
+
+DSA uses the ``rtnl_link_ops`` mechanism (with a "dsa" ``kind``) to allow
+changing the DSA master of a user port. The ``IFLA_DSA_MASTER`` u32 netlink
+attribute contains the ifindex of the master device that handles each slave
+device. The DSA master must be a valid candidate based on firmware node
+information, or a LAG interface which contains only slaves which are valid
+candidates.
+
+Using iproute2, the following manipulations are possible:
+
+  .. code-block:: sh
+
+    # See the DSA master in current use
+    ip -d link show dev swp0
+        (...)
+        dsa master eth0
+
+    # Static CPU port distribution
+    ip link set swp0 type dsa master eth1
+    ip link set swp1 type dsa master eth0
+    ip link set swp2 type dsa master eth1
+    ip link set swp3 type dsa master eth0
+
+    # CPU ports in LAG, using explicit assignment of the DSA master
+    ip link add bond0 type bond mode balance-xor && ip link set bond0 up
+    ip link set eth1 down && ip link set eth1 master bond0
+    ip link set swp0 type dsa master bond0
+    ip link set swp1 type dsa master bond0
+    ip link set swp2 type dsa master bond0
+    ip link set swp3 type dsa master bond0
+    ip link set eth0 down && ip link set eth0 master bond0
+    ip -d link show dev swp0
+        (...)
+        dsa master bond0
+
+    # CPU ports in LAG, relying on implicit migration of the DSA master
+    ip link add bond0 type bond mode balance-xor && ip link set bond0 up
+    ip link set eth0 down && ip link set eth0 master bond0
+    ip link set eth1 down && ip link set eth1 master bond0
+    ip -d link show dev swp0
+        (...)
+        dsa master bond0
+
+Notice that in the case of CPU ports under a LAG, the use of the
+``IFLA_DSA_MASTER`` netlink attribute is not strictly needed, but rather, DSA
+reacts to the ``IFLA_MASTER`` attribute change of its present master (``eth0``)
+and migrates all user ports to the new upper of ``eth0``, ``bond0``. Similarly,
+when ``bond0`` is destroyed using ``RTM_DELLINK``, DSA migrates the user ports
+that were assigned to this interface to the first physical DSA master which is
+eligible, based on the firmware description (it effectively reverts to the
+startup configuration).
+
+In a setup with more than 2 physical CPU ports, it is therefore possible to mix
+static user to CPU port assignment with LAG between DSA masters. It is not
+possible to statically assign a user port towards a DSA master that has any
+upper interfaces (this includes LAG devices - the master must always be the LAG
+in this case).
+
+Live changing of the DSA master (and thus CPU port) affinity of a user port is
+permitted, in order to allow dynamic redistribution in response to traffic.
+
+Physical DSA masters are allowed to join and leave at any time a LAG interface
+used as a DSA master; however, DSA will reject a LAG interface as a valid
+candidate for being a DSA master unless it has at least one physical DSA master
+as a slave device.
diff --git a/Documentation/networking/dsa/dsa.rst b/Documentation/networking/dsa/dsa.rst
index d742ba6bd211..a94ddf83348a 100644
--- a/Documentation/networking/dsa/dsa.rst
+++ b/Documentation/networking/dsa/dsa.rst
@@ -303,6 +303,20 @@ These frames are then queued for transmission using the master network device
 Ethernet switch will be able to process these incoming frames from the
 management interface and deliver them to the physical switch port.
 
+When using multiple CPU ports, it is possible to stack a LAG (bonding/team)
+device between the DSA slave devices and the physical DSA masters. The LAG
+device is thus also a DSA master, but the LAG slave devices continue to be DSA
+masters as well (just with no user port assigned to them; this is needed for
+recovery in case the LAG DSA master disappears). Thus, the data path of the LAG
+DSA master is used asymmetrically. On RX, the ``ETH_P_XDSA`` handler, which
+calls ``dsa_switch_rcv()``, is invoked early (on the physical DSA master;
+LAG slave). Therefore, the RX data path of the LAG DSA master is not used.
+On the other hand, TX takes place linearly: ``dsa_slave_xmit`` calls
+``dsa_enqueue_skb``, which calls ``dev_queue_xmit`` towards the LAG DSA master.
+The latter calls ``dev_queue_xmit`` towards one physical DSA master or the
+other, and in both cases, the packet exits the system through a hardware path
+towards the switch.
+
 Graphical representation
 ------------------------
 
@@ -629,6 +643,24 @@ Switch configuration
   PHY cannot be found. In this case, probing of the DSA switch continues
   without that particular port.
 
+- ``port_change_master``: method through which the affinity (association used
+  for traffic termination purposes) between a user port and a CPU port can be
+  changed. By default all user ports from a tree are assigned to the first
+  available CPU port that makes sense for them (most of the times this means
+  the user ports of a tree are all assigned to the same CPU port, except for H
+  topologies as described in commit 2c0b03258b8b). The ``port`` argument
+  represents the index of the user port, and the ``master`` argument represents
+  the new DSA master ``net_device``. The CPU port associated with the new
+  master can be retrieved by looking at ``struct dsa_port *cpu_dp =
+  master->dsa_ptr``. Additionally, the master can also be a LAG device where
+  all the slave devices are physical DSA masters. LAG DSA masters also have a
+  valid ``master->dsa_ptr`` pointer, however this is not unique, but rather a
+  duplicate of the first physical DSA master's (LAG slave) ``dsa_ptr``. In case
+  of a LAG DSA master, a further call to ``port_lag_join`` will be emitted
+  separately for the physical CPU ports associated with the physical DSA
+  masters, requesting them to create a hardware LAG associated with the LAG
+  interface.
+
 PHY devices and link management
 -------------------------------
 
@@ -1095,9 +1127,3 @@ capable hardware, but does not enforce a strict switch device driver model. On
 the other DSA enforces a fairly strict device driver model, and deals with most
 of the switch specific. At some point we should envision a merger between these
 two subsystems and get the best of both worlds.
-
-Other hanging fruits
---------------------
-
-- allowing more than one CPU/management interface:
-  http://comments.gmane.org/gmane.linux.network/365657
-- 
2.34.1