Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
To:     Koba Ko <koba.ko@canonical.com>,
        "David S. Miller" <davem@davemloft.net>,
        Jakub Kicinski <kuba@kernel.org>, netdev@vger.kernel.org,
        linux-kernel@vger.kernel.org
References: <20210608032207.2923574-1-koba.ko@canonical.com>
From:   Heiner Kallweit <hkallweit1@gmail.com>
Subject: Re: [PATCH] [v2] r8169: Use PHY_POLL when RTL8106E enable ASPM
Message-ID: <84eb168e-58ff-0350-74e2-c55249eb258c@gmail.com>
Date:   Tue, 8 Jun 2021 10:00:38 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <20210608032207.2923574-1-koba.ko@canonical.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 08.06.2021 05:22, Koba Ko wrote:
> For RTL8106E, it's a Fast-ethernet chip.
> If ASPM is enabled, the link chang interrupt wouldn't be triggered
> immediately and must wait a very long time to get link change interrupt.
> Even the link change interrupt isn't triggered, the phy link is already
> established.
> 
> Use PHY_POLL to watch the status of phy link and disable
> the link change interrupt when ASPM is enabled on RTL8106E.
> 
> v2: Instead use PHY_POLL and identify 8106E by RTL_GIGA_MAC_VER_39.
> 

Still the issue description doesn't convince me that it's a hw bug
with the respective chip version. What has been stated so far:

1. (and most important) Issue doesn't occur in mainline because ASPM
   is disabled in mainline for r8169. Issue occurs only with a
   downstream kernel with ASPM enabled for r8169.

2. Issue occurs only with ASPM L1.1 not disabled, even though this chip
   version doesn't support L1 sub-states. Just L0s/L1 don't trigger
   the issue.
   The NIC doesn't announce L1.1 support, therefore PCI core won't
   enable L1 sub-states on the PCIe link between NIC and upstream
   PCI bridge.

3. Issue occurs only with a GBit-capable link partner. 100MBit link
   partners are fine. Not clear whether issue occurs with a specific
   Gbit link partner only or with GBit-capable link partners in general.

4. Only link-up interrupt is affected. Not link-down and not interrupts
   triggered by other interrupt sources.

5. Realtek couldn't confirm that there's such a hw bug on RTL8106e.

One thing that hasn't been asked yet:
Does issue occur always if you re-plug the cable? Or only on boot?
I'm asking because in the dmesg log you attached to the bugzilla issue
the following looks totally ok.

[   61.651643] r8169 0000:01:00.0 enp1s0: Link is Down
[   63.720015] r8169 0000:01:00.0 enp1s0: Link is Up - 100Mbps/Full - flow control rx/tx
[   66.685499] r8169 0000:01:00.0 enp1s0: Link is Down

> Signed-off-by: Koba Ko <koba.ko@canonical.com>
> ---
>  drivers/net/ethernet/realtek/r8169_main.c | 21 +++++++++++++++++++--
>  1 file changed, 19 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
> index 2c89cde7da1e..a59cbaef2839 100644
> --- a/drivers/net/ethernet/realtek/r8169_main.c
> +++ b/drivers/net/ethernet/realtek/r8169_main.c
> @@ -4914,6 +4914,19 @@ static const struct dev_pm_ops rtl8169_pm_ops = {
>  
>  #endif /* CONFIG_PM */
>  
> +static int rtl_phy_poll_quirk(struct rtl8169_private *tp)
> +{
> +	struct pci_dev *pdev = tp->pci_dev;
> +
> +	if (!pcie_aspm_enabled(pdev))

That's the wrong call. According to what you said earlier you want to
check for L1 sub-states, not for ASPM in general.

> +		return 0;
> +
> +	if (tp->mac_version == RTL_GIGA_MAC_VER_39)
> +		return 1;
> +
> +	return 0;
> +}
> +
>  static void rtl_wol_shutdown_quirk(struct rtl8169_private *tp)
>  {
>  	/* WoL fails with 8168b when the receiver is disabled. */
> @@ -4991,7 +5004,10 @@ static const struct net_device_ops rtl_netdev_ops = {
>  
>  static void rtl_set_irq_mask(struct rtl8169_private *tp)
>  {
> -	tp->irq_mask = RxOK | RxErr | TxOK | TxErr | LinkChg;
> +	tp->irq_mask = RxOK | RxErr | TxOK | TxErr;
> +
> +	if (!rtl_phy_poll_quirk(tp))
> +		tp->irq_mask |= LinkChg;
>  
>  	if (tp->mac_version <= RTL_GIGA_MAC_VER_06)
>  		tp->irq_mask |= SYSErr | RxOverflow | RxFIFOOver;
> @@ -5085,7 +5101,8 @@ static int r8169_mdio_register(struct rtl8169_private *tp)
>  	new_bus->name = "r8169";
>  	new_bus->priv = tp;
>  	new_bus->parent = &pdev->dev;
> -	new_bus->irq[0] = PHY_MAC_INTERRUPT;
> +	new_bus->irq[0] =
> +		(rtl_phy_poll_quirk(tp) ? PHY_POLL : PHY_MAC_INTERRUPT);
>  	snprintf(new_bus->id, MII_BUS_ID_SIZE, "r8169-%x", pci_dev_id(pdev));
>  
>  	new_bus->read = r8169_mdio_read_reg;
>