Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Sandor Bodo-Merle <sbodomerle@gmail.com>
To:     =?UTF-8?q?Rafa=C5=82=20Mi=C5=82ecki?= <rafal@milecki.pl>,
        Broadcom internal kernel review list 
        <bcm-kernel-feedback-list@broadcom.com>,
        "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>,
        Felix Fietkau <nbd@openwrt.org>, netdev@vger.kernel.org,
        linux-kernel@vger.kernel.org
Cc:     Sandor Bodo-Merle <sbodomerle@gmail.com>
Subject: [PATCH] net: bgmac:`Fix a BUG triggered by wrong bytes_compl
Date:   Mon,  8 Aug 2022 19:39:39 +0200
Message-Id: <20220808173939.193804-1-sbodomerle@gmail.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk

On one of our machines we got:

kernel BUG at lib/dynamic_queue_limits.c:27!\r\n
Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM\r\n
CPU: 0 PID: 1166 Comm: irq/41-bgmac Tainted: G        W  O    4.14.275-rt132 #1\r\n
Hardware name: BRCM XGS iProc\r\n
task: ee3415c0 task.stack: ee32a000\r\n
PC is at dql_completed+0x168/0x178\r\n
LR is at bgmac_poll+0x18c/0x6d8\r\n
pc : [<c03b9430>]    lr : [<c04b5a18>]    psr: 800a0313\r\n
sp : ee32be14  ip : 000005ea  fp : 00000bd4\r\n
r10: ee558500  r9 : c0116298  r8 : 00000002\r\n
r7 : 00000000  r6 : ef128810  r5 : 01993267  r4 : 01993851\r\n
r3 : ee558000  r2 : 000070e1  r1 : 00000bd4  r0 : ee52c180\r\n
Flags: Nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none\r\n
Control: 12c5387d  Table: 8e88c04a  DAC: 00000051\r\n
Process irq/41-bgmac (pid: 1166, stack limit = 0xee32a210)\r\n
Stack: (0xee32be14 to 0xee32c000)\r\n
be00:                                              ee558520 ee52c100 ef128810\r\n
be20: 00000000 00000002 c0116298 c04b5a18 00000000 c0a0c8c4 c0951780 00000040\r\n
be40: c0701780 ee558500 ee55d520 ef05b340 ef6f9780 ee558520 00000001 00000040\r\n
be60: ffffe000 c0a56878 ef6fa040 c0952040 0000012c c0528744 ef6f97b0 fffcfb6a\r\n
be80: c0a04104 2eda8000 c0a0c4ec c0a0d368 ee32bf44 c0153534 ee32be98 ee32be98\r\n
bea0: ee32bea0 ee32bea0 ee32bea8 ee32bea8 00000000 c01462e4 ffffe000 ef6f22a8\r\n
bec0: ffffe000 00000008 ee32bee4 c0147430 ffffe000 c094a2a8 00000003 ffffe000\r\n
bee0: c0a54528 00208040 0000000c c0a0c8c4 c0a65980 c0124d3c 00000008 ee558520\r\n
bf00: c094a23c c0a02080 00000000 c07a9910 ef136970 ef136970 ee30a440 ef136900\r\n
bf20: ee30a440 00000001 ef136900 ee30a440 c016d990 00000000 c0108db0 c012500c\r\n
bf40: ef136900 c016da14 ee30a464 ffffe000 00000001 c016dd14 00000000 c016db28\r\n
bf60: ffffe000 ee21a080 ee30a400 00000000 ee32a000 ee30a440 c016dbfc ee25fd70\r\n
bf80: ee21a09c c013edcc ee32a000 ee30a400 c013ec7c 00000000 00000000 00000000\r\n
bfa0: 00000000 00000000 00000000 c0108470 00000000 00000000 00000000 00000000\r\n
bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000\r\n
bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000\r\n
[<c03b9430>] (dql_completed) from [<c04b5a18>] (bgmac_poll+0x18c/0x6d8)\r\n
[<c04b5a18>] (bgmac_poll) from [<c0528744>] (net_rx_action+0x1c4/0x494)\r\n
[<c0528744>] (net_rx_action) from [<c0124d3c>] (do_current_softirqs+0x1ec/0x43c)\r\n
[<c0124d3c>] (do_current_softirqs) from [<c012500c>] (__local_bh_enable+0x80/0x98)\r\n
[<c012500c>] (__local_bh_enable) from [<c016da14>] (irq_forced_thread_fn+0x84/0x98)\r\n
[<c016da14>] (irq_forced_thread_fn) from [<c016dd14>] (irq_thread+0x118/0x1c0)\r\n
[<c016dd14>] (irq_thread) from [<c013edcc>] (kthread+0x150/0x158)\r\n
[<c013edcc>] (kthread) from [<c0108470>] (ret_from_fork+0x14/0x24)\r\n
Code: a83f15e0 0200001a 0630a0e1 c3ffffea (f201f0e7) \r\n

The issue seems similar to commit 90b3b339364c ("net: hisilicon: Fix a BUG
trigered by wrong bytes_compl") and potentially introduced by commit
b38c83dd0866 ("bgmac: simplify tx ring index handling").

If there is an RX interrupt between setting ring->end
and netdev_sent_queue() we can hit the BUG_ON as bgmac_dma_tx_free()
can miscalculate the queue size while called from bgmac_poll().

The machine which triggered the BUG runs a v4.14 RT kernel - but the issue
seems present in mainline too.

Fixes: b38c83dd0866 ("bgmac: simplify tx ring index handling")
Signed-off-by: Sandor Bodo-Merle <sbodomerle@gmail.com>
---
 drivers/net/ethernet/broadcom/bgmac.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/broadcom/bgmac.c b/drivers/net/ethernet/broadcom/bgmac.c
index 2dfc1e32bbb3..93580484a3f4 100644
--- a/drivers/net/ethernet/broadcom/bgmac.c
+++ b/drivers/net/ethernet/broadcom/bgmac.c
@@ -189,8 +189,8 @@ static netdev_tx_t bgmac_dma_tx_add(struct bgmac *bgmac,
 	}
 
 	slot->skb = skb;
-	ring->end += nr_frags + 1;
 	netdev_sent_queue(net_dev, skb->len);
+	ring->end += nr_frags + 1;
 
 	wmb();
 
-- 
2.36.1