From:   Vladimir Oltean <vladimir.oltean@nxp.com>
To:     =?iso-8859-1?Q?Cl=E9ment_L=E9ger?= <clement.leger@bootlin.com>
CC:     "David S. Miller" <davem@davemloft.net>,
        Jakub Kicinski <kuba@kernel.org>,
        Rob Herring <robh+dt@kernel.org>,
        Claudiu Manoil <claudiu.manoil@nxp.com>,
        Alexandre Belloni <alexandre.belloni@bootlin.com>,
        "UNGLinuxDriver@microchip.com" <UNGLinuxDriver@microchip.com>,
        Andrew Lunn <andrew@lunn.ch>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "devicetree@vger.kernel.org" <devicetree@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Thomas Petazzoni <thomas.petazzoni@bootlin.com>,
        Denis Kirjanov <dkirjanov@suse.de>,
        Julian Wiedmann <jwi@linux.ibm.com>
Subject: Re: [PATCH net-next v3 4/4] net: ocelot: add FDMA support
Thread-Topic: [PATCH net-next v3 4/4] net: ocelot: add FDMA support
Thread-Index: AQHX4ur0oKFzimZda0eyFp+wSrHWQKwXeIeAgAK1KwCAAJzpAA==
Date:   Mon, 29 Nov 2021 17:40:39 +0000
Message-ID: <20211129174038.gptbivgmbqzrrgtz@skbuf>
References: <20211126172739.329098-1-clement.leger@bootlin.com>
 <20211126172739.329098-5-clement.leger@bootlin.com>
 <20211127145805.75qh2vim7c5m5hjd@skbuf> <20211129091902.0112eb17@fixe.home>
In-Reply-To: <20211129091902.0112eb17@fixe.home>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?u0jHV5pAS4Qu8GAkrHKAsgc4zsXZwcM/W8eYa5cIqkC/X36mQKr15WMP+P?=
 =?iso-8859-1?Q?EtXAsB2ZRTsqtkFA/B2C4lXRmdX44iMalzWZSfe5UEAe2xQ0CV9ctbdzlp?=
 =?iso-8859-1?Q?fzNngwJfVkPZDr9iZdIv3AzGjyWBFrD/kzi45xYCN5TdGkduvVAFsf4WHD?=
 =?iso-8859-1?Q?8vTf8d15cuOnjHwgwMG9CBKgz94W2d7TTVQMmkBxpyfKs1L7KJ8PJ3vHX1?=
 =?iso-8859-1?Q?Za7JqCM2nBnwJ0ilRooUPaQ9ZaZ7dCpvKMfv5wDiAh8Zyh2IkqrybNS6Kw?=
 =?iso-8859-1?Q?kT4OYFlZUesEFlE46FEniQo1logJ5dxQOm1rrgWBWvtOeTG6c75fGS67NJ?=
 =?iso-8859-1?Q?GhGXP0nYd4382EmESRZTfpsaoQEpj7AGlIEu0Wt1+1RrE2K0ysnXOqMRfz?=
 =?iso-8859-1?Q?gS6lCOcMRJlYnBfd8JtulYcFX2LE6XEn+5eL9GVGD4JTbZKQ6coyfzqFEK?=
 =?iso-8859-1?Q?VEZa0+jzN8xUCxEe6OyRg0787Q9T4l0uhEM5sAR1fd9m9+8F9oEIYTBVe0?=
 =?iso-8859-1?Q?WUsk47enGQOZ/gPg73CwdH6VFObwth9dbP0um8iGlAMAbiancD/uKhWwPG?=
 =?iso-8859-1?Q?UIPVQLt0oWPIjn05EudM+PcL6so6aRXeFXX6ym32gPvXuW0RbEamc0Z5dr?=
 =?iso-8859-1?Q?KS0Mvq0iqkJ+gSXUVM1pvzKvOwcUxswdCFIvU6JfzwwrXS+MukN5Zp1oko?=
 =?iso-8859-1?Q?BAy9RfoC3QhpgGPstGpoqyz9E77UUtUZIsCAD1+wShzY/IAgdiauzpexHp?=
 =?iso-8859-1?Q?LfJon/nDlW2MBkOcZUhdZ0Ycq298euuOMVhifIyvxIsQpyo+/WJ4NHSqxa?=
 =?iso-8859-1?Q?tabsDSD4WpBgk6R6qrJYvhWhtykNHrnPVTjRQ3lqWMgMfPJLbpk5d2cRZJ?=
 =?iso-8859-1?Q?DsuUEoKnkKBPph5OIGhP/3SkOB/asMwKOw/yGQoKSL2OxYdS5XLw/7keAR?=
 =?iso-8859-1?Q?WW+2lp1908OR3ifhJOn+4AgSHPMCBN5ppM+91/O9+H4DXkjqH/68gaW2oz?=
 =?iso-8859-1?Q?tlXVg4YC5xYjQigfbXAXRC3gAskOpubH6P3qdn1UXYHRFRwc27hSDRkh/e?=
 =?iso-8859-1?Q?7CvNdvh3JIlJz214FEee2XuPM7hTYqB1//97tSJy1IGbz9EERgdSjG73lD?=
 =?iso-8859-1?Q?1rLeGuRtZ+vfnrvIqiZnNzY0VTC4p+wiZQpht18Y2WIL3mgrcDMB7h0d4t?=
 =?iso-8859-1?Q?BrtuLG2yBVqgJUqWKdMkT6ja07t6U+H9Cr55ytV24Grs59kJZs7fd1s395?=
 =?iso-8859-1?Q?ST6R+8QxHKPBO7VqWNh98dC8y3tZRN5gYCQUTOy37T5KKh3O7hw+JaTvYY?=
 =?iso-8859-1?Q?W3NMpOENrYVuoyU8kSB9cmAlGaj2T7CVQEG1/pyskth0SPjwJB6s5LGuG9?=
 =?iso-8859-1?Q?BdJMdWVGDZTCOaTQwTNgF48iUG0nhQvtqmmo+qtzzA3X4rAcWydiHZaLIp?=
 =?iso-8859-1?Q?+GRSblLKpl28nhjKl5UbyJkLNKUajyrTnX81v19LRrVqSvFhKD25cHpS39?=
 =?iso-8859-1?Q?/+1VWqO7ZMcloiZk7JZ7X/x/QranORd7yGjXI3gr35xobCj3EvD85rxMKa?=
 =?iso-8859-1?Q?fbiyMUwUDzVnSu2CkpsjU+8Lekrl1/5X7TbvIG+tOD1zivqpliVkoB3mtu?=
 =?iso-8859-1?Q?Odwre2+rARGGMitl2bkHyX3gCQbHNC6GVGlQjtpjzPoVQ01Hu8imVTnuAo?=
 =?iso-8859-1?Q?bWAKJ19vZ/ozmYYTCKM=3D?=
Content-Type: text/plain; charset="iso-8859-1"
Content-ID: <8469D342D3D67D49AAA34A6D424CD819@eurprd04.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: VI1PR04MB5136.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a7e47792-3302-419d-df6d-08d9b35f5be3
X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Nov 2021 17:40:39.3042
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: hQsoWgES2we4kgsDNvkwvlxE7lTyP6LsczDJRHRbqJurKH6pr3VZWpJTP2LKmUlVcqjDXkmcr3gkE0QvBQMoVg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1PR04MB7470
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Nov 29, 2021 at 09:19:02AM +0100, Cl=E9ment L=E9ger wrote:
> > I'm not sure why you're letting the hardware grind to a halt first,
> > before refilling? I think since the CPU is the bottleneck anyway, you
> > can stop the extraction channel at any time you want to refill.
> > A constant stream of less data might be better than a bursty one.
> > Or maybe I'm misunderstanding some of the details of the hardware.
>=20
> Indeed, I can stop the extraction channel but that does not seems a
> good idea to stop the channel in a steady state. At least that's what I
> thought since it will make the receive "window" non predictable. Not
> sure how well it will play with various protocol but I will try
> implementing the refill we talked previously (ie when there an
> available threshold is reached).
(...)
> > I don't understand why you restart the injection channel from the TX
> > confirmation interrupt. It raised the interrupt to tell you that it hit
> > a NULL LLP because there's nothing left to send. If you restart it now =
and
> > no other transmission has happened in the meantime, won't it stop again=
?
>=20
> Actually, it is only restarted if there is some pending packets to
> send. With this hardware, packets can't be added while the FDMA is
> running and it must be stopped everytime we want to add a packet to the
> list. To avoid that, in the TX path, if the FDMA is stopped, we set the
> llp of the packet to NULL and start the chan. However, if the FDMA TX
> channel is running, we don't stop it, we simply add the next packets to
> the ring. However, the FDMA will stop on the previous NULL LLP. So when
> we hit a LLP, we might not be at the end of the list. This is why the
> next check verifies if we hit a NULL LLP and if there is still some
> packet to send.=20

Oh, is that so? That would be pretty odd if the hardware is so dumb that
it doesn't detect changes made to an LLP on the go.

The manual has this to say, and I'm not sure how to interpret it:

| It is possible to update an active channels LLP pointer and pointers in
| the DCB chains. Before changing pointers software must schedule the
| channel for disabling (by writing FDMA_CH_DISABLE.CH_DISABLE[ch]) and
| then wait for the channel to set FDMA_CH_SAFE.CH_SAFE[ch]. When the
| pointer update is complete, soft must re-activate the channel by setting
| FDMA_CH_ACTIVATE.CH_ACTIVATE[ch]. Setting activate will cancel the
| deactivate-request, or if the channel has disabled itself in the
| meantime, it will re activate the channel.

So it is possible to update an active channel's LLP pointer, but not
while it's active? Thank you very much!

If true, this will severely limit the termination performance one is
able to obtain with this switch, even with a faster CPU and PCIe.

> > > +void ocelot_fdma_netdev_init(struct ocelot_fdma *fdma, struct net_de=
vice *dev)
> > > +{
> > > +	dev->needed_headroom =3D OCELOT_TAG_LEN;
> > > +	dev->needed_tailroom =3D ETH_FCS_LEN; =20
> >=20
> > The needed_headroom is in no way specific to FDMA, right? Why aren't yo=
u
> > doing it for manual register-based injection too? (in a separate patch =
ofc)
>=20
> Actually, If I switch to page based ring, This won't be useful anymore
> because the header will be written directly in the page and not anymore
> directly in the skb header.

I don't understand this comment. You set up the needed headroom and
tailroom netdev variables to avoid reallocation on TX, not for RX.
And you use half page buffers for RX, not for TX.

> > I can't help but think how painful it is that with a CPU as slow as
> > yours, insult over injury, you also need to check for each packet
> > whether the device tree had defined the "fdma" region or not, because
> > you practically keep two traffic I/O implementations due to that sole
> > reason. I think for the ocelot switchdev driver, which is strictly for
> > MIPS CPUs embedded within the device, it should be fine to introduce a
> > static key here (search for static_branch_likely in the kernel).
>=20
> I thinked about it *but* did not wanted to add a key since it would be
> global. However, we could consider that there is always only one
> instance of the driver and indeed a static key is an option.
> Unfortunately, I'm not sure this will yield any noticeable performance
> improvement.

What is the concern with a static key in this driver, exactly?=