Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Sean Anderson <sean.anderson@seco.com>
Subject: Re: [PATCH net-next v3 10/47] net: phylink: Adjust link settings
 based on rate adaptation
To:     Andrew Lunn <andrew@lunn.ch>
Cc:     "David S . Miller" <davem@davemloft.net>,
        Jakub Kicinski <kuba@kernel.org>,
        Madalin Bucur <madalin.bucur@nxp.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Eric Dumazet <edumazet@google.com>,
        linux-arm-kernel@lists.infradead.org,
        Russell King <linux@armlinux.org.uk>,
        linux-kernel@vger.kernel.org,
        Alexandru Marginean <alexandru.marginean@nxp.com>,
        Heiner Kallweit <hkallweit1@gmail.com>,
        Vladimir Oltean <olteanv@gmail.com>
References: <20220715215954.1449214-1-sean.anderson@seco.com>
 <20220715215954.1449214-11-sean.anderson@seco.com> <YtMc2qYWKRn2PxRY@lunn.ch>
 <4172fd87-8e51-e67d-bf86-fdc6829fa9b3@seco.com> <YtNoW8bJdWPzX3Cq@lunn.ch>
Message-ID: <41d87b16-e0ea-5b29-6ecc-8e90f906d366@seco.com>
Date:   Mon, 18 Jul 2022 12:29:17 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.10.0
In-Reply-To: <YtNoW8bJdWPzX3Cq@lunn.ch>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SjM5aGgwck1CY1R1a1BFSFNKWFJuWXp3RHUzQTRwa015bUpQVzlFNnNjSElQ?=
 =?utf-8?B?MUxoV3BLaVNRQXFGbXdveEJIUU1rYmhYRkQrQmY0Y0hFUW02TXNDZ0lFWTZw?=
 =?utf-8?B?RmF4WDNsQVN2UVhqS0pLK3o0aXVkaUhZUzRONGJrZXpyTHRCMHdjOHAxeG1J?=
 =?utf-8?B?R041OXFTWHF6ekZuNUwyUnYyYkhEc28xLy9WT2pjMU9FYWl4SytmZk9zOGt4?=
 =?utf-8?B?YU9ZcDRiWDFGMEN5dXN6bXp0T3gwOWgweUlwT1FPbGNobWlLc3FTY1FYb29a?=
 =?utf-8?B?WjhSNk4vQzNJS0tTUE1uNWNxUEZXOXMrYWZoSWRBV0wyWkl5d0ZpYmpmS0NE?=
 =?utf-8?B?ODlJY3U1Rml4Rkk4M3pzSExrdDFIdnpSejBKWkVpQVFkUW5UWndlUDZnRnFV?=
 =?utf-8?B?SlhKWGRKdXpmT1prL2lZL2NBRy8wUENsaTlGanRpVjZJQ05kOVplWDdUZllQ?=
 =?utf-8?B?TkpQcWxocnBGaG9PRW9Fd1JqYWt4QnF0anVPcGNuOTFaSzdDeFluVThYckdF?=
 =?utf-8?B?WUo0eTFNVUF3cSt5NUZoSVlXU0RYb0RRU25Oa1F2bFd5WEZYUEpoNVBCV2ts?=
 =?utf-8?B?emgyQVhoUXArQ0JucEp1REVqcWc5TzMzdzVUT2UrNjJ0VnIvWDBrL3p3OWhr?=
 =?utf-8?B?Q0pod2VoNmFOTjR5YVNBYXgrbFBOZTh0cGJWMjhOaXZrQlRheDhJNHVZWERm?=
 =?utf-8?B?L05od2ZmN0lqWVltSm9EQ1I5a0w0ckxEQkZiTHRpWFp3Nm1Jb2tYdVJiWjlK?=
 =?utf-8?B?ZnkzN1FWb09tNHpaVmdweTkzK2JTeVI3ajRydnkycXRsekxYcFh5ME04RWgv?=
 =?utf-8?B?MkZ2OWJxZWdOWHB1MVZYazZPVmpZVjBXc3kxb1VtMkpWNmQ5RTBQYjhFRFJM?=
 =?utf-8?B?aEc5QTlBbmtMc0QyM1JXaUJVYVZEUmdZalRPTHFFc05QWmk4RjE5YmZsUTVE?=
 =?utf-8?B?Q2YxazRxdWEwZ05GdWRQZUxNSVNEL0NZNC8vUjEzZEJvMGh5RUlkakFJcEUz?=
 =?utf-8?B?a29Hd0tHNXVROGRrTWpuSjhPWWRsWWZ6cVhZdW9WVTJkd3IyL3JuTkJYaExv?=
 =?utf-8?B?Ri9iNVY2VC95TmpvQ3FpeGM5ZTBkT3ErdW5WUU1GWGg3U1llVTE0enN6NzYr?=
 =?utf-8?B?VFVIYStZM3NrbmlSUUxuS1pTeThyV3p5M0wyTmNLc21pWlB4ZWhPWkRSN0lH?=
 =?utf-8?B?Zk55U0VLRG5ZVUo0T2lVQkNGWFd3UWszLzFVYzBnMFo4ZDlaNXdBL3g4cThz?=
 =?utf-8?B?L1ZuUjJPRVN5N2laSDBIdHVVcXNrS0NBU1RWUVFVT1R4dGFJTGFZaVhucFhP?=
 =?utf-8?B?enBST3YrVjZ2R3hxampsZHFTUHBKL2pyczRtc2xMcHhwd1BNbUZHd1ZpNDhB?=
 =?utf-8?B?enhHMGN0VGY2T0Ryd1ArKzA5UGVrUHI4SXIwbUY4N1JId2ZqdjZNeFpKSHBN?=
 =?utf-8?B?dEREZG1CNXMzUkxtdjNlSGhvUXhvS2pNVjA1OW96RWdoMmltSDAxSVdlTGNR?=
 =?utf-8?B?MXRrNzhPaEpPejZMSzhSMkRFYk9TU3BCVWV2ZlYzTGtwOTM3RW5KWjlSeitX?=
 =?utf-8?B?VXBhUTlwd3VjOXlLUE91Z01tb0pxZ3lRT0cyRUYzQWJwWGo2NWsxcER3QVgz?=
 =?utf-8?B?czAzSEVqcjVVa1JUQlQyL3RYc2NXRTBVd3p3bi9EYW9QRHN1RVhGMlVFQ0tZ?=
 =?utf-8?B?Mm8vK0dOMmNOa2toMG5vTFN6SnF4UUwwL3F3MzVMU2tBQjc2R0RQYVFwTUFn?=
 =?utf-8?B?QzVlZ1NOWlU3ZXJIYktuaUtmZlhKVnBlVFh0NG40MXhpTDJzUFJkbzJyTGt3?=
 =?utf-8?B?MFNoT21yTlRtVGhDeXp3Y0hCNFoxOVJadGxGM2d3UXlyREE3K2hJN3FONkdt?=
 =?utf-8?B?OVNZbkxDQ1EwbWhGUXUxT2p3U1FmVmJkYXN2MmIrZmg3LzErU3hCblRNNXBX?=
 =?utf-8?B?RDErMHR4a2JzN3UrM28zNjlDYWx5U3VJa1VEZkoxRkI3a0Y3MXJkR2tNWEVR?=
 =?utf-8?B?M21sVmsxZWVVaGwxUEtBclU5cnc4Mys0cVBaWmdvOVpBQjlDL2ZabWhDMWNW?=
 =?utf-8?B?NUtncXBCeGVFZko2cDRaTFY1UCt0S2VoQndrYU1QM3RjUUxabS8wVzh2ck5s?=
 =?utf-8?B?ZWxRblNYT0lUWFY1aExOQ0VLRWJGU0FVTS8wYWxYbmVlM3gwQ1lycGg1MG1q?=
 =?utf-8?B?R0E9PQ==?=
X-OriginatorOrg: seco.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8de6ed8c-e9ce-4d02-452a-08da68daac38
X-MS-Exchange-CrossTenant-AuthSource: DB7PR03MB4972.eurprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jul 2022 16:29:23.0218
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: bebe97c3-6438-442e-ade3-ff17aa50e733
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: j4CWhPj+7SLelQi03mj2U918v5GbPQsz/Q5+dxIxYos2W4nNHIv3lMeL3FevfoYCM7ifFORwYqfM7YZHmdlYEQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB7PR03MB5097
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 7/16/22 9:39 PM, Andrew Lunn wrote:
>> > I would not do this. If the requirements for rate adaptation are not
>> > fulfilled, you should turn off rate adaptation.
>> > 
>> > A MAC which knows rate adaptation is going on can help out, by not
>> > advertising 10Half, 100Half etc. Autoneg will then fail for modes
>> > where rate adaptation does not work.
>> 
>> OK, so maybe it is better to phylink_warn here. Something along the
>> lines of "phy using pause-based rate adaptation, but duplex is %s".
> 
> You say 1/2 duplex simply does not work with rate adaptation.

It doesn't work with pause-based rate adaptation. This is because we can't
enable pause frames in half duplex (see phy_get_pause). I don't know if this
is a technical limitation (or something else), but presumably there exists a
MAC out there which can't enable pause frames unless it's in full-duplex mode.

> So i
> would actually return -EINVAL at the point the MAC indicates what
> modes it supports if there is a 1/2 duplex mode in the list.

Well, half duplex is still valid if we are at the full line rate. This is more
of a sanity check on what we get back from the phy. That is, we should never
get anything but full duplex if the phy indicates that pause-based rate
adaptation is being performed. So maybe this should live in phy_read_status?

And of course, CRS-based adaptation requires half-duplex (or a MAC which
respects CRS in full-duplex mode).

>> 
>> > The MAC should also be declaring what sort of pause it supports, so
>> > disable rate adaptation if it does not have async pause.
>> 
>> That's what we do in the previous patch.
>> 
>> The problem is that rx_pause and tx_pause are resolved based on our
>> advertisement and the link partner's advertisement. However, the link
>> partner may not support pause frames at all. In that case, we will get
>> rx_pause and tx_pause as false. However, we still want to enable rx_pause,
>> because we know that the phy will be emitting pause frames. And of course
>> the user can always force disable pause frames anyway through ethtool.
> 
> Right, so we need a table somewhere in the documentation listing the
> different combinations and what should happen.

OK, so first here's table 28B-3 (e.g. linkmode_resolve_pause):

Local device  Link partner  Local resolution Partner resolution
============= ============= ================ ==================
PAUSE ASM_DIR PAUSE ASM_DIR Transmit Receive Transmit   Receive
===== ======= ===== ======= ======== ======= ========   =======
    0       0     X       X        N       N        N         N
    0       1     0       X        N       N        N         N
    0       1     1       0        N       N        N         N
    0       1     1       1        Y       N        N         Y
    1       0     0       X        N       N        N         N
    1       X     1       X        Y       Y        Y         Y
    1       1     0       0        N       N        N         N
    1       1     0       1        N       Y        Y         N

And now here's the same table, but assuming that we have a local phy
performing rate adaptation

Local device  Link partner  Local resolution Partner resolution
============= ============= ================ ==================
PAUSE ASM_DIR PAUSE ASM_DIR Transmit Receive Transmit   Receive
===== ======= ===== ======= ======== ======= ========   =======
    0       0     X       X        N       N        N         N # Broken
    0       1     0       X        N       N        N         N # Broken
    0       1     1       0        N       N        N         N # Broken
    0       1     1       1        Y       N        N         Y # Broken
    1       0     0       X        ?       ?        N         N # Semi-broken
    1       X     1       X        Y       Y        Y         Y
    1       1     0       0        N       Y        N         N
    1       1     0       1        N       Y        Y         N

The rows marked as "Broken" don't have local receive pause enabled.
These should never occur, since we can detect that the local MAC doesn't
support pause reception and disable advertisement of pause-based
rate-adapted modes.

On the row marked as "Semi-broken", the local MAC supports only
symmetric pause, and the link partner doesn't support pause. We're not
supposed to send pause frames, so we disable pause, but this breaks rate
adaptation. In this case, we could renegotiate with rate-adapted modes
disabled. Alternatively, we could just decline to advertise rate-adapted
modes for symmetric-pause MACs. This avoids the semi-broken line above,
but also prevents the line below from using rate adaptation.

> If the MAC does not support rx_pause, rate adaptation is turned off.
>> If the negotiation results in no rx_pause, force it on anyway with
> Pause based adaptation. If ethtool turns pause off, turn off rate
> adaptation.
> 
> Does 802.3 say anything about this?

Only IPG-based and CRS-based rate adaptation are defined in 802.3.

> We might also want to add an additional state to the ethtool get for
> pause, to indicate rx_pause is enabled because of rate adaptation, not
> because of autoneg.

Probably a good idea.

--Sean