Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Tomas Melin <tomas.melin@vaisala.com>
To:     claudiu.beznea@microchip.com
Cc:     Nicolas.Ferre@microchip.com, davem@davemloft.net,
        linux-kernel@vger.kernel.org, netdev@vger.kernel.org
Subject: Re: [PATCH v3] net: macb: restart tx after tx used bit read
Date:   Wed, 23 Mar 2022 10:08:20 +0200
Message-Id: <20220323080820.137579-1-tomas.melin@vaisala.com>
In-Reply-To: <1545040937-6583-1-git-send-email-claudiu.beznea@microchip.com>
References: <1545040937-6583-1-git-send-email-claudiu.beznea@microchip.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?tZjl4Y2dSo1P4ECI5xAcAVjBWduZEdhIX717BoCEomQAKio3V74ncDzY2GbV?=
 =?us-ascii?Q?GITt772ZeiN26Jtr9bnwAiwQt3EILWywROUjzuIn1lM1d5hpGkKfHykmIbw3?=
 =?us-ascii?Q?oU2K9YvNhVhpJqSPwDOt5JqybJqXHMCaMcZ6b21o9RfzzmZ30WewCR2wqZ2W?=
 =?us-ascii?Q?X6W8yUvX+e32Qg5sxFlVivTKe5Dx24jEuEiWtMkUyRI0PxgQr/frQn06xeeF?=
 =?us-ascii?Q?hCgwuNXuL3RB3cNhfQunl5FMOW3MGZZKGFc3b7eT8g4dT3Z70j4wsygVvtyh?=
 =?us-ascii?Q?s8RY0eN8Y7AsXoEi1GB5Y/WJtzel7FTClSLoyCzngW/tyPIXmGOLHOTaAMla?=
 =?us-ascii?Q?C1JappW4UG/JnrKZuGknMrHm13/L4S1B5Puk7HiL5z0+PXzg/BJo3Sf9jwoO?=
 =?us-ascii?Q?t+kLLCR12EjhqWtYKyu47678E7GQVB1rN87/pslGwIH2535gOmr1wvFzsUVn?=
 =?us-ascii?Q?VRreTNa0JlAz2cKDVwnEjYdZXBr5BQQ0Sbzbs0rhVdLj/D+KlVfMeG+Vv3Rn?=
 =?us-ascii?Q?6weSvpz7povvzOIP+JjJiNGI3diFzHf4w4zU1qrIX70NjpLMssN8KoaomX7G?=
 =?us-ascii?Q?snGPFYaD+tse4ueSxVt/JesfCZE+fbTQbnFi2IIrXT0bnBEGTeAqcFsXs7rA?=
 =?us-ascii?Q?nlDQi8jme1QoQcU6o7DjKxu1bJbzUuQl16TWn0CRAGIqdK/fDWLrkiyCAklv?=
 =?us-ascii?Q?DtDW4LohEeyOv+G/4yME3YB8MLhmHGmPjfKlFNpQMg1uthFEt+L5TmSJF8dA?=
 =?us-ascii?Q?xRTgsrG9EfHSE2uGI6sljLRvbJynhIirK5Xxe/CacSQFwhgGo/JydlhluF3K?=
 =?us-ascii?Q?6L6CxzKxxMDxhiuQDxEMKuUekGnFka4Socs3wpPXGuhm9sN1Kq4N9EXJnS7Q?=
 =?us-ascii?Q?twIKcZOHmL7S4B/46xFXmzPCouCkQaIZRFvqpUWOJ/Q5+1u+8gyKHHO8eUr/?=
 =?us-ascii?Q?ATCKqLDlGERV4+i+w/yaVHdFINawJmwfIzLGmU/XnphsfS7PmGOdHTnGvCdY?=
 =?us-ascii?Q?hHdgIbwHrQPvk8A0TlTXkpN2lH5XmN22e5oWw7oa2I2ewQGZ20ReeZ2EpMZC?=
 =?us-ascii?Q?5e6uay8ZrBv+zZPGB+SVF1Xp86aqQ9n5Jw+MXQxfnI2oQtuFCWPDx5ApIGZ3?=
 =?us-ascii?Q?sP9xyW5WGEOjqadvnFXDNE5xweUWsUgei/36cs5d7ZADZyZJntXhK8PqwAl/?=
 =?us-ascii?Q?gpcW7jzhMlAvbAEaeJS7HNhsuVqVBjxWtiQJ+EMUETXmpBx5tXZ0HXSSAFQK?=
 =?us-ascii?Q?wytqSukCxLJjHcAii+ClToJEM35fnhsd/4Ve54A7HzAJZYq4Mpt0C1Qt6vNQ?=
 =?us-ascii?Q?w5tfH3HXlWx5A5kHEFMSaZt4/rbcyxPFN1n+uGREfaixtt0BLIX6A3iGX5/H?=
 =?us-ascii?Q?67ysZdmy7I/XyccdhN3IMDpRl80aE74izqCnbtETcT/f65zbblq5eoc8YfcU?=
 =?us-ascii?Q?O4X9S0vvhaJti+9gB+D6K9awlAd/wz8sJrjs0L6ASxGtReVuO5Hhsw=3D=3D?=
X-OriginatorOrg: vaisala.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b8fbc32c-8ccb-4236-2702-08da0ca45ffa
X-MS-Exchange-CrossTenant-AuthSource: HE1PR0602MB3625.eurprd06.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Mar 2022 08:08:55.3089
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 6d7393e0-41f5-4c2e-9b12-4c2be5da5c57
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 6tsxSxyWbIFt5zRY3PsAJIAVspmHP7H3FhwugircE8fB1pTWYzkGPgYQrUPM9GCsjNYb7d67FPFwTFt57WQgCQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR06MB8214
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,
        T_SCC_BODY_TEXT_LINE,T_SPF_TEMPERROR autolearn=no autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi,

> From: <Claudiu.Beznea@microchip.com>
> To: <Nicolas.Ferre@microchip.com>, <davem@davemloft.net>
> Cc: <netdev@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
> 	<Claudiu.Beznea@microchip.com>
> Subject: [PATCH v3] net: macb: restart tx after tx used bit read
> Date: Mon, 17 Dec 2018 10:02:42 +0000	[thread overview]
> Message-ID: <1545040937-6583-1-git-send-email-claudiu.beznea@microchip.com> (raw)
> 
> From: Claudiu Beznea <claudiu.beznea@microchip.com>
> 
> On some platforms (currently detected only on SAMA5D4) TX might stuck
> even the pachets are still present in DMA memories and TX start was
> issued for them. This happens due to race condition between MACB driver
> updating next TX buffer descriptor to be used and IP reading the same
> descriptor. In such a case, the "TX USED BIT READ" interrupt is asserted.
> GEM/MACB user guide specifies that if a "TX USED BIT READ" interrupt
> is asserted TX must be restarted. Restart TX if used bit is read and
> packets are present in software TX queue. Packets are removed from software
> TX queue if TX was successful for them (see macb_tx_interrupt()).
> 
> Signed-off-by: Claudiu Beznea <claudiu.beznea@microchip.com>

On Xilinx Zynq the above change can cause infinite interrupt loop leading 
to CPU stall. Seems timing/load needs to be appropriate for this to happen, and currently
with 1G ethernet this can be triggered normally within minutes when running stress tests
on the network interface.

The events leading up to the interrupt looping are similar as the issue described in the
commit message. However in our case, restarting TX does not help at all. Instead
the controller is stuck on the queue end descriptor generating endless TX_USED           
interrupts, never breaking out of interrupt routine.

Any chance you remember more details about in which situation restarting TX helped for
your use case? was tx_qbar at the end of frame or stopped in middle of frame?

thanks,
Tomas Melin


> ---
> 
> Changes in v3:
> - remove "inline" keyword
> 
> Changes in v2:
> - use "static inline" instead of "inline static" for macb_tx_restart()
> 
>  drivers/net/ethernet/cadence/macb_main.c | 21 ++++++++++++++++++++-
>  1 file changed, 20 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ethernet/cadence/macb_main.c b/drivers/net/ethernet/cadence/macb_main.c
> index 1d86b4d5645a..f920230386ee 100644
> --- a/drivers/net/ethernet/cadence/macb_main.c
> +++ b/drivers/net/ethernet/cadence/macb_main.c
> @@ -61,7 +61,8 @@
>  #define MACB_TX_ERR_FLAGS	(MACB_BIT(ISR_TUND)			\
>  					| MACB_BIT(ISR_RLE)		\
>  					| MACB_BIT(TXERR))
> -#define MACB_TX_INT_FLAGS	(MACB_TX_ERR_FLAGS | MACB_BIT(TCOMP))
> +#define MACB_TX_INT_FLAGS	(MACB_TX_ERR_FLAGS | MACB_BIT(TCOMP)	\
> +					| MACB_BIT(TXUBR))
>  
>  /* Max length of transmit frame must be a multiple of 8 bytes */
>  #define MACB_TX_LEN_ALIGN	8
> @@ -1312,6 +1313,21 @@ static void macb_hresp_error_task(unsigned long data)
>  	netif_tx_start_all_queues(dev);
>  }
>  
> +static void macb_tx_restart(struct macb_queue *queue)
> +{
> +	unsigned int head = queue->tx_head;
> +	unsigned int tail = queue->tx_tail;
> +	struct macb *bp = queue->bp;
> +
> +	if (bp->caps & MACB_CAPS_ISR_CLEAR_ON_WRITE)
> +		queue_writel(queue, ISR, MACB_BIT(TXUBR));
> +
> +	if (head == tail)
> +		return;
> +
> +	macb_writel(bp, NCR, macb_readl(bp, NCR) | MACB_BIT(TSTART));
> +}
> +
>  static irqreturn_t macb_interrupt(int irq, void *dev_id)
>  {
>  	struct macb_queue *queue = dev_id;
> @@ -1369,6 +1385,9 @@ static irqreturn_t macb_interrupt(int irq, void *dev_id)
>  		if (status & MACB_BIT(TCOMP))
>  			macb_tx_interrupt(queue);
>  
> +		if (status & MACB_BIT(TXUBR))
> +			macb_tx_restart(queue);
> +
>  		/* Link change detection isn't possible with RMII, so we'll
>  		 * add that if/when we get our hands on a full-blown MII PHY.
>  		 */
> -- 
> 2.7.4
>