Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Hariprasad Kelam <hkelam@marvell.com>
To:     Maxim Mikityanskiy <maxtram95@gmail.com>
CC:     "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "kuba@kernel.org" <kuba@kernel.org>,
        "davem@davemloft.net" <davem@davemloft.net>,
        "pabeni@redhat.com" <pabeni@redhat.com>,
        "edumazet@google.com" <edumazet@google.com>,
        Sunil Kovvuri Goutham <sgoutham@marvell.com>,
        Linu Cherian <lcherian@marvell.com>,
        Geethasowjanya Akula <gakula@marvell.com>,
        Jerin Jacob Kollanukkaran <jerinj@marvell.com>,
        Subbaraya Sundeep Bhatta <sbhatta@marvell.com>,
        "jhs@mojatatu.com" <jhs@mojatatu.com>,
        "xiyou.wangcong@gmail.com" <xiyou.wangcong@gmail.com>,
        "jiri@resnulli.us" <jiri@resnulli.us>,
        "saeedm@nvidia.com" <saeedm@nvidia.com>,
        "richardcochran@gmail.com" <richardcochran@gmail.com>,
        "tariqt@nvidia.com" <tariqt@nvidia.com>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
        "hariprasad.netdev@gmail.com" <hariprasad.netdev@gmail.com>
Subject: Re: [net-next Patch v2 4/5] octeontx2-pf: Add devlink support to
 configure TL1 RR_PRIO
Thread-Topic: [net-next Patch v2 4/5] octeontx2-pf: Add devlink support to
 configure TL1 RR_PRIO
Thread-Index: AQHZLPEZRqmv6MllqUaKMRPDHkfLgw==
Date:   Fri, 20 Jan 2023 17:03:20 +0000
Message-ID: <PH0PR18MB4474DBEF155EFA4DA6BA5B10DEC59@PH0PR18MB4474.namprd18.prod.outlook.com>
References: <20230118105107.9516-1-hkelam@marvell.com>
 <20230118105107.9516-5-hkelam@marvell.com> <Y8hYlYk/7FfGdfy8@mail.gmail.com>
 <PH0PR18MB4474FCEAC4FA5907CAC17011DEC59@PH0PR18MB4474.namprd18.prod.outlook.com>
 <Y8qZNhUgsdOMavC4@mail.gmail.com>
In-Reply-To: <Y8qZNhUgsdOMavC4@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?HMy4hMq2PpQVz03V3RUm3q6tzysPb/9RBJtkId2U7xkWjJqOADh+R/O0hssX?=
 =?us-ascii?Q?pS/3RxK87FY0+U7MSrkI/rDICQyxvmLKKpO5LpKFQlLHy/EKLX4WZS5xaFK5?=
 =?us-ascii?Q?kPQ/fbcc5ETXukmjXYeHAeKheIESbAgJllv7ixwRLj2S7Jv594CSqjQ2tVxO?=
 =?us-ascii?Q?EIoveaeGwnEN8V1ht0go1CjRAghzBG9nb7LwXUX7Kxi/OlII9x/jPRdhxnB4?=
 =?us-ascii?Q?v+2JMT7B8ffx2fD48z4iX1TLMmiMT4+kLyGNo20CRhugR37AwRVBzBcRb0yw?=
 =?us-ascii?Q?WYgipUJpIDEtq8YtRTXughy0JRQwIZx3niYWMPldH7Y8OVzN+v9oxtT5IJ3X?=
 =?us-ascii?Q?ofGp4OQZI/ymoygwoYo80G+z8rgwKPUIEWrvadTTVY/atEUQzNNaNe/SnYYH?=
 =?us-ascii?Q?rR/66MM8npqDUerM9ArCa+Ym1S9BJrro+LBzTC8qUMDS06Ja2AvOEWWHrwLb?=
 =?us-ascii?Q?mh0XfG/KK3LdI/zD0ZzV18qornsvrnFvMlrRWXGTE/3k5kTB7H9JCx+5BUHH?=
 =?us-ascii?Q?usFNrkI8yyZYFrdJ0ma2ZzBQEtq89EHBhsDkkpJG+d04EVxUctuZXxDFAqAg?=
 =?us-ascii?Q?8s5IvJjuoqKAe3p1ons06lKA6V1M41LIWtPs9aL6cBOoTxTgA3c9gpmpx0WB?=
 =?us-ascii?Q?zr47ka4gYxhCszl0oCup/NQc6+3+76zKAa/pRaZlHRZiID4RiU/xQVloO2sJ?=
 =?us-ascii?Q?286IeGnyb26yCHLYNEpotRnqXrYWEwHRZcsJYpZXkjKJXOtXns2WY3tUXDmh?=
 =?us-ascii?Q?kliAjund6WYCpdSf6j6TZaf4LBxexCHB4I1SRtR0Y5Gu33G6kv9REaninmdA?=
 =?us-ascii?Q?SxE8a0upnJCs3+9ilgNeQ7KPpD3rGlkqivUieWz667RqmeYRM+1O4L3QzFQ1?=
 =?us-ascii?Q?vZNbluFSAb6RIGnuA8eLfxTXOsMnErRx5KEDggJjqBhvrzzeHSbbImP0zM5x?=
 =?us-ascii?Q?vkPwvPyRyoHEn0McMdSzZcB3i8moy9ynFB3rW1XZQ2Az/RYDxFhJw9MA4y1U?=
 =?us-ascii?Q?kvebxsjZfgWhsRmduTKmPg8ZEazKeFEdBELT4czf018nhcPB9ipIaY8FXfHZ?=
 =?us-ascii?Q?ZEYpoNl+P2C0ntWWHH4aYHvjI60BM6p5hNa2cDLxATsIkEvKXxq0f+AnA0QS?=
 =?us-ascii?Q?VnyL1NGQ6Xpml5FPwi/WrbRoxrlHHZmr/kB3S9QwBP32Jel9RlY3ntWX502X?=
 =?us-ascii?Q?0iwhRzLe7gHrHps8QmukYyo63yPmPi7iYAnHvDnxRwaKlWeVoaVs3anxsXdd?=
 =?us-ascii?Q?QKq5rlh2YNVltNtFQVFNi109IlAS3yp4vm/WRApSHnwe2LzEOi4kUuJLndG/?=
 =?us-ascii?Q?1qfLGnVjN7/V0RFdVaPkoh0kUz1Q0J0N28mIpB1WXv3lKDF8xd4ativpXJ6B?=
 =?us-ascii?Q?9oSVTDjYRME5SUUWXgYoRDMgfGtC6ky0OTStH99QSGe56kQ4xrweyuMhU4hf?=
 =?us-ascii?Q?suSKYJBf4a+ug7MhSQnHaWnJ0uBwl8eeeOjLqN9F1AQxOmNnw+nUoiJZQpM/?=
 =?us-ascii?Q?kNsc6gwIWMHz5jGFdqtMZHYbGwPHCGCJME2jlSLWgd2vqoUNqesIhFJt/c35?=
 =?us-ascii?Q?Oz8G/1NBRKYMviPHiYqwMYh5lMlHMLIPIdvIl/6P?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: marvell.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR18MB4474.namprd18.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 727161af-69cd-4dad-68a9-08dafb083bbd
X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jan 2023 17:03:20.6769
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: PGHQt9i/g6j+6bdZlhDT24Wh+CNzPpgnRImCpLl8pK1xWRB1CedAxNAYC1VGuCk+BZPwOJcKml2yk+EezOJ8Jw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR18MB4630
X-Proofpoint-GUID: mlulinFln0092EA8HiuWks5p5uxJZNlK
X-Proofpoint-ORIG-GUID: mlulinFln0092EA8HiuWks5p5uxJZNlK
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1
 definitions=2023-01-20_09,2023-01-20_01,2022-06-22_01
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,RCVD_IN_DNSWL_LOW,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On Fri, Jan 20, 2023 at 08:50:16AM +0000, Hariprasad Kelam wrote:
>=20
> On Wed, Jan 18, 2023 at 04:21:06PM +0530, Hariprasad Kelam wrote:
> > All VFs and PF netdev shares same TL1 schedular, each interface PF=20
> > or VF will have different TL2 schedulars having same parent TL1. The=20
> > TL1 RR_PRIO value is static and PF/VFs use the same value to=20
> > configure its
> > TL2 node priority in case of DWRR children.
> >=20
> > This patch adds support to configure TL1 RR_PRIO value using devlink.
> > The TL1 RR_PRIO can be configured for each PF. The VFs are not=20
> > allowed to configure TL1 RR_PRIO value. The VFs can get the RR_PRIO=20
> > value from the mailbox NIX_TXSCH_ALLOC response parameter aggr_lvl_rr_p=
rio.
>=20
> I asked this question under v1, but didn't get an answer, could you shed =
some light?
>=20
> "Could you please elaborate how these priorities of Transmit Levels are r=
elated to HTB priorities? I don't seem to understand why something has to b=
e configured with devlink in addition to HTB.
>=20
> SMQ (send meta-descriptor queue) and MDQ (meta-descriptor queue) are the =
first transmit levels.
> Each send queue is mapped with SMQ.
> =20
> As mentioned in cover letter, each egress packet needs to traverse all tr=
ansmit levels starting from TL5 to TL1.

Yeah, I saw that, just some details about your hardware which might be obvi=
ous to you aren't so clear to me...

Do these transmit levels map to "layers" of HTB hierarchy? Does it look lik=
e this, or is my understanding completely wrong?

TL1                 [HTB root node]
                   /               \
TL2          [HTB node]         [HTB node]
            /          \             |
TL3    [HTB node]  [HTB node]   [HTB node]
...                       ...

Transmit levels to HTB mapping is correct.


> This applies to non-QOS Send queues as well.
> =20
>                        SMQ/MDQ --> TL4 -->TL3 -->TL2 -->TL1
>=20
> By default non QOS queues use a default hierarchy  with round robin prior=
ity.=20
> To avoid conflict with QOS tree priorities, with devlink user can choose =
round-robin priority before Qos tree formation.

So, this priority that you set with devlink is basically a weight of unclas=
sified (default) traffic for round robin between unclassified and classifie=
d traffic, right? I.e. you have two hierarchies (one for HTB, another for n=
on-QoS queue), and you do DWRR between them, according to this priority?


 Not exactly, In the given scenario where  multiple vfs are attached to PF =
netdev.
 each VF unclassified traffic forms a hierarchy and PF also forms a hierarc=
hy for unclassified traffic.
=20
Now traffic from these all tress(multiple vfs and PFs) are aggregated at TL=
1. HW performs DWRR among them since these TL2 queues (belonging to each pf=
 and vf netdevs) will be configured with the same priority by the driver.

Currently, this priority is hard coded. Now we are providing this as a conf=
igurable value to the user.

Now if a user adds a HTB node, this will have strict priority at TL2 level =
since DWRR priority is different this traffic won't be affected by DWRR unc=
lassified traffic.

Thanks,
Hariprasad k


> BTW, why did you remove the paragraphs with an example and a limitation?
> I think they are pretty useful.
>=20
> Ok , removed them accidentally will correct in the next version.
>=20
> Another question unanswered under v1 was:
>=20
> "Is there any technical difficulty or hardware limitation preventing=20
> from implementing modifications?" (TC_HTB_NODE_MODIFY)
>=20
> There is no hardware limitation, we are currently implementing it.  once =
it's implemented we will submit for review.

Great, that's nice to hear, looking forward to it.