Received-SPF: pass (google.com: domain of linux-kernel+bounces-186230-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Wed, 22 May 2024 14:37:50 +0200
From: Andrew Lunn <andrew@lunn.ch>
To: Larry Chiu <larry.chiu@realtek.com>
Cc: Justin Lai <justinlai0215@realtek.com>,
	"kuba@kernel.org" <kuba@kernel.org>,
	"davem@davemloft.net" <davem@davemloft.net>,
	"edumazet@google.com" <edumazet@google.com>,
	"pabeni@redhat.com" <pabeni@redhat.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"jiri@resnulli.us" <jiri@resnulli.us>,
	"horms@kernel.org" <horms@kernel.org>,
	Ping-Ke Shih <pkshih@realtek.com>
Subject: Re: [PATCH net-next v19 01/13] rtase: Add pci table supported in
 this module
Message-ID: <7aab03ba-d8ed-4c9c-8bfd-b2bbed0a922d@lunn.ch>
References: <20240517075302.7653-1-justinlai0215@realtek.com>
 <20240517075302.7653-2-justinlai0215@realtek.com>
 <d840e007-c819-42df-bc71-536328d4f5d7@lunn.ch>
 <e5d7a77511f746bdb0b38b6174ef5de4@realtek.com>
 <97e30c5f-1656-46d0-b06c-3607a90ec96f@lunn.ch>
 <f9133a36bbae41138c3080f8f6282bfd@realtek.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <f9133a36bbae41138c3080f8f6282bfd@realtek.com>

On Wed, May 22, 2024 at 04:43:11AM +0000, Larry Chiu wrote:
> 
> > On Tue, May 21, 2024 at 06:20:04AM +0000, Larry Chiu wrote:
> > >
> > > >> + *  Below is a simplified block diagram of the chip and its relevant
> > interfaces.
> > > >> + *
> > > >> + *               *************************
> > > >> + *               *                       *
> > > >> + *               *  CPU network device   *
> > > >> + *               *                       *
> > > >> + *               *   +-------------+     *
> > > >> + *               *   |  PCIE Host  |     *
> > > >> + *               ***********++************
> > > >> + *                          ||
> > > >> + *                         PCIE
> > > >> + *                          ||
> > > >> + *      ********************++**********************
> > > >> + *      *            | PCIE Endpoint |             *
> > > >> + *      *            +---------------+             *
> > > >> + *      *                | GMAC |                  *
> > > >> + *      *                +--++--+  Realtek         *
> > > >> + *      *                   ||     RTL90xx Series  *
> > > >> + *      *                   ||                     *
> > > >> + *      *     +-------------++----------------+    *
> > > >> + *      *     |           | MAC |             |    *
> > > >> + *      *     |           +-----+             |    *
> > > >> + *      *     |                               |    *
> > > >> + *      *     |     Ethernet Switch Core      |    *
> > > >> + *      *     |                               |    *
> > > >> + *      *     |   +-----+           +-----+   |    *
> > > >> + *      *     |   | MAC |...........| MAC |   |    *
> > > >> + *      *     +---+-----+-----------+-----+---+    *
> > > >> + *      *         | PHY |...........| PHY |        *
> > > >> + *      *         +--++-+           +--++-+        *
> > > >> + *      *************||****************||***********
> > > >> + *
> > > >> + *  The block of the Realtek RTL90xx series is our entire chip
> > > >> + architecture,
> > > >> + *  the GMAC is connected to the switch core, and there is no PHY in
> > between.
> > > >
> > > >Given this architecture, this driver cannot be used unless there is a switch
> > > >driver as well. This driver is nearly ready to be merged. So what are your
> > > >plans for the switch driver? Do you have a first version you can post? That
> > > >will reassure us you do plan to release a switch driver, and not use a SDK in
> > > >userspace.
> > > >
> > > >        Andrew
> > >
> > > Hi Andrew,
> > > This GMAC is configured after the switch is boot-up and does not require a
> > > switch driver to work.
> > 
> > But if you cannot configure the switch, it is pointless passing the switch
> > packets. The Linux architecture is that Linux needs to be able to control the
> > switch somehow. There needs to be a driver with the switchdev API on its
> > upper side which connects it to the Linux network stack. Ideally the lower
> > side of this driver can directly write switch registers. Alternatively it can make
> > some sort of RPC to firmware which configures the switch.
> > 
> > Before committing this MAC driver, we will want to be convinced there is a
> > switchdev driver for the switch.
> > 
> >         Andrew
> 
> 
> I know what you mean.
> But actually this GMAC works like a NIC connected to an Ethernet Switch not a 
> management port, its packets communicating with other ports.

Linux has two different models for switches.

The first is switchdev. Linux has a netdev per port of the switch, and
use you those netdev's to manage the switch, just as if they are
individual NICs.

The second is very, very old, since the beginning of Ethernet
switches. The cable comes out of the machine and plugs into the
switch. Linux has no idea there is a switch there, the switch is just
part of the magic if networking. This also means Linux cannot manage
the switch, it is a different box, a different administration domain.

The second model does not really work here. The switch is not in
another box at the end of a cable. It is integrated into the SoC!
 
> The PCIe Endpoint is a multi-function device, the other function is used to 
> control the switch register, we are still working on where to put this driver in 
> Linux. We thought it should be separated into different device drivers, or you 
> think we should register two pcie functions in this driver.

Look at the architecture of other switch drivers. There are two broad
categories.

1) Pure switchdev drivers, e.g. mellanox, sparx5, prestera. There is
one driver which provides both the netdev interfaces per port, and
implements the switchdev API for managing the switch.

2) DSA + switchdev, e.g. mv88e6xxx, rtl8365, starfigher2, etc. These
use a conventional NIC to provide the conduit to pass packets to the
switch. These packets have additional headers, added by a tag driver,
indicating which port a packet should go out. And there is a switch
driver, which makes use of the DSA framework to manage the switch. DSA
provides the netdev per port.

This is actually something i ask you about with version 1 of the
patches. I've forget what your answer was, and we concentrated on
getting your code up to mainline quality. Now it is time to go back to
that question.

How do you control where a packet passed over this GMAC NIC goes
within the switch? Is there an additional header? Are their fields in
the DMA descriptor?

If your hardware is DSA like, you can write another driver which binds
to a different PCI function. If however you use DMA descriptors, you
need a pure switchdev driver, one driver which binds to multiple PCI
functions.

	Andrew