Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
IronPort-SDR: KSg3GLb8szZk24q82Ncn/9o5zNf0nVnIxMwGZ9xYKExGgjPq3rKboyG8XLFvNytikeTnBbvN71
 bxxxbkudEd1ZTgBykagqEf9WF7hPPblxvS4vXroAXzC53Sa2b+tHqooRyIOFqaKEGlRvSin1hZ
 QAj+OLlabkTUEiPiL6fXuGTtoCs8e5Vr3aUuWzIdFAk+C9FcaacNQlzR+3GiOklg9vCqUUG142
 iYJI4rcuyxrrZawGTbuZj91tHi5TYHGMnAfJLtEqyJ6v6278PbU+0p7hjFicv5oepSjwrPODUp
 ZZU=
From:   <Nicolas.Ferre@microchip.com>
To:     <linux-arm-kernel@lists.infradead.org>, <netdev@vger.kernel.org>,
        <Claudiu.Beznea@microchip.com>, <harini.katakam@xilinx.com>,
        <f.fainelli@gmail.com>, <linux@armlinux.org.uk>
CC:     <andrew@lunn.ch>, <alexandre.belloni@bootlin.com>,
        <sergio.prado@e-labworks.com>, <pthombar@cadence.com>,
        <antoine.tenart@bootlin.com>, <linux-kernel@vger.kernel.org>,
        <michal.simek@xilinx.com>, <davem@davemloft.net>
Subject: Re: [PATCH 4/5] net: macb: WoL support for GEM type of Ethernet
 controller
Thread-Topic: [PATCH 4/5] net: macb: WoL support for GEM type of Ethernet
 controller
Thread-Index: AQHWF7XmxtnWGGqiwUmnXwZJVbFung==
Date:   Tue, 21 Apr 2020 08:21:48 +0000
Message-ID: <8fcf4a8a-362c-a71f-c99e-be9500db7371@microchip.com>
References: <cover.1587058078.git.nicolas.ferre@microchip.com>
 <56bb7a742093cec160c4465c808778a14b2607e7.1587058078.git.nicolas.ferre@microchip.com>
 <61762f4b-03fa-5484-334e-8515eed485e2@microchip.com>
In-Reply-To: <61762f4b-03fa-5484-334e-8515eed485e2@microchip.com>
Accept-Language: en-US
Content-Language: en-US
user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.7.0
received-spf: None (protection.outlook.com: microchip.com does not designate
 permitted sender hosts)
Content-Type: text/plain; charset="Windows-1252"
Content-ID: <6B9155F7FF221448B790C7012F8DF301@namprd11.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 21bd8544-b71b-4c82-0de0-08d7e5cd0954
X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Apr 2020 08:21:48.2576
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 3f4057f3-b418-4d4e-ba84-d55b4e897d88
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: akiQvFzyjaZcZM1jtYnS5OKfWhRZRt0YU3+39eAO+ymcmzbN/D4InCwJksi0Rb++k1Crh0AqUw8PJ502QDZEMLXcplWSg9Rn5s8Fj11HeVU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR11MB3454
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 17/04/2020 at 19:14, Nicolas Ferre wrote:
> On 16/04/2020 at 19:44, nicolas.ferre@microchip.com wrote:
>> From: Nicolas Ferre <nicolas.ferre@microchip.com>
>>
>> Adapt the Wake-on-Lan feature to the Cadence GEM Ethernet controller.
>> This controller has different register layout and cannot be handled by
>> previous code.
>> We disable completely interrupts on all the queues but the queue 0.
>> Handling of WoL interrupt is done in another interrupt handler
>> positioned depending on the controller version used, just between
>> suspend() and resume() calls.
>> It allows to lower pressure on the generic interrupt hot path by
>> removing the need to handle 2 tests for each IRQ: the first figuring out
>> the controller revision, the second for actually knowing if the WoL bit
>> is set.
>>
>> Queue management in suspend()/resume() functions inspired from RFC patch
>> by Harini Katakam <harinik@xilinx.com>, thanks!
>>
>> Signed-off-by: Nicolas Ferre <nicolas.ferre@microchip.com>
>> ---
>>    drivers/net/ethernet/cadence/macb.h      |   3 +
>>    drivers/net/ethernet/cadence/macb_main.c | 121 ++++++++++++++++++++--=
-
>>    2 files changed, 109 insertions(+), 15 deletions(-)
>=20
> [..]
>=20
>> @@ -4534,23 +4564,56 @@ static int __maybe_unused macb_suspend(struct de=
vice *dev)
>>    	struct macb_queue *queue =3D bp->queues;
>>    	unsigned long flags;
>>    	unsigned int q;
>> +	int err;
>>   =20
>>    	if (!netif_running(netdev))
>>    		return 0;
>>   =20
>>    	if (bp->wol & MACB_WOL_ENABLED) {
>> -		macb_writel(bp, IER, MACB_BIT(WOL));
>> -		macb_writel(bp, WOL, MACB_BIT(MAG));
>> -		enable_irq_wake(bp->queues[0].irq);
>> -		netif_device_detach(netdev);
>> -	} else {
>> -		netif_device_detach(netdev);
>> +		spin_lock_irqsave(&bp->lock, flags);
>> +		/* Flush all status bits */
>> +		macb_writel(bp, TSR, -1);
>> +		macb_writel(bp, RSR, -1);
>>    		for (q =3D 0, queue =3D bp->queues; q < bp->num_queues;
>> -		     ++q, ++queue)
>> -			napi_disable(&queue->napi);
>> -		rtnl_lock();
>> -		phylink_stop(bp->phylink);
>> -		rtnl_unlock();
>> +		     ++q, ++queue) {
>> +			/* Disable all interrupts */
>> +			queue_writel(queue, IDR, -1);
>> +			queue_readl(queue, ISR);
>> +			if (bp->caps & MACB_CAPS_ISR_CLEAR_ON_WRITE)
>> +				queue_writel(queue, ISR, -1);
>> +		}
>> +		/* Change interrupt handler and
>> +		 * Enable WoL IRQ on queue 0
>> +		 */
>> +		if (macb_is_gem(bp)) {
>> +			devm_free_irq(dev, bp->queues[0].irq, bp->queues);
>> +			err =3D devm_request_irq(dev, bp->queues[0].irq, gem_wol_interrupt,
>> +					       IRQF_SHARED, netdev->name, bp->queues);
>> +			if (err) {
>> +				dev_err(dev,
>> +					"Unable to request IRQ %d (error %d)\n",
>> +					bp->queues[0].irq, err);
>> +				return err;
>> +			}
>> +			queue_writel(bp->queues, IER, GEM_BIT(WOL));
>> +			gem_writel(bp, WOL, MACB_BIT(MAG));
>> +		} else {
>> +			queue_writel(bp->queues, IER, MACB_BIT(WOL));
>> +			macb_writel(bp, WOL, MACB_BIT(MAG));
>> +		}
>> +		spin_unlock_irqrestore(&bp->lock, flags);
>> +
>> +		enable_irq_wake(bp->queues[0].irq);
>> +	}
>> +
>> +	netif_device_detach(netdev);
>> +	for (q =3D 0, queue =3D bp->queues; q < bp->num_queues;
>> +	     ++q, ++queue)
>> +		napi_disable(&queue->napi);
>> +
>> +	if (!(bp->wol & MACB_WOL_ENABLED)) {
>> +		phy_stop(netdev->phydev);
>> +		phy_suspend(netdev->phydev);
>=20
> Bug here: you must read:
>=20
> 		rtnl_lock();
> 		phylink_stop(bp->phylink);
> 		rtnl_unlock();
>=20
> Instead of the 2 previous lines. I'll correct in v2.
>=20
> Sorry for the regression.
>=20
>=20
>>    		spin_lock_irqsave(&bp->lock, flags);
>>    		macb_reset_hw(bp);
>>    		spin_unlock_irqrestore(&bp->lock, flags);
>> @@ -4575,20 +4638,48 @@ static int __maybe_unused macb_resume(struct dev=
ice *dev)
>=20
> [..]
> BTW: I have issue having a real resume event from the phy with this
> series. I'm investigating that but didn't find anything for now.
>=20
> Observation #1: when the WoL is not enabled, I don't have link issue.
> But the path in suspend/resume is far more intrusive in phy state.
>=20
> Observation #2: when WoL is enabled, I need to do a full ifdown/ifup
> sequence for gain access again to the link:
>=20
> ip link show eth0
> 2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast
> state DOWN mode DEFAULT group default qlen 1000
>       link/ether 54:10:ec:be:50:b0 brd ff:ff:ff:ff:ff:ff
>=20
> ifdown eth0 && ifup eth0
>=20
> ip link show eth0
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> state UP mode DEFAULT group default qlen 1000
>       link/ether 54:10:ec:be:50:b0 brd ff:ff:ff:ff:ff:ff
>=20
> Observation #3: I didn't experience this behavior while playing with the
> WoL on my 4.19 kernel before porting to Mainline.

I've reviewed this series to fix this last issues. It's was a=20
combination of runtime_pm not handled properly and a mix of=20
netif_carrier_* call with phylink calls not well positioned nor balanced=20
between suspend and resume.

I have a v2 series that I'm preparing for today: Harini, I prefer to=20
post it now so it could avoid that you hit the same issues as me while=20
testing on your platform.

Best regards,
    Nicolas


--=20
Nicolas Ferre