Received-SPF: pass (google.com: domain of linux-kernel+bounces-45181-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
From: Alex Elder <elder@linaro.org>
To: davem@davemloft.net,
	edumazet@google.com,
	kuba@kernel.org,
	pabeni@redhat.com
Cc: mka@chromium.org,
	andersson@kernel.org,
	quic_cpratapa@quicinc.com,
	quic_avuyyuru@quicinc.com,
	quic_jponduru@quicinc.com,
	quic_subashab@quicinc.com,
	elder@kernel.org,
	netdev@vger.kernel.org,
	linux-arm-msm@vger.kernel.org,
	linux-kernel@vger.kernel.org
Subject: [PATCH net-next 2/7] net: ipa: begin simplifying TX queue stop
Date: Tue, 30 Jan 2024 13:22:59 -0600
Message-Id: <20240130192305.250915-3-elder@linaro.org>
In-Reply-To: <20240130192305.250915-1-elder@linaro.org>
References: <20240130192305.250915-1-elder@linaro.org>
Precedence: bulk
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

There are a number of flags used in the IPA driver to attempt to
manage race conditions that can occur between runtime resume and
netdev transmit.  If we disable TX before requesting power, we can
avoid these races entirely, simplifying things considerably.

This patch implements the main change, disabling transmit always in
the net_device->ndo_start_xmit() callback, then re-enabling it again
whenever we find power is active (or when we drop the skb).

The patches that follow will refactor the "old" code to the point
that most of it can be eliminated.

Signed-off-by: Alex Elder <elder@linaro.org>
---
 drivers/net/ipa/ipa_modem.c | 41 +++++++++++++++++++++++-----------
 drivers/net/ipa/ipa_power.c | 44 ++++++++++++++++++++++---------------
 2 files changed, 54 insertions(+), 31 deletions(-)

diff --git a/drivers/net/ipa/ipa_modem.c b/drivers/net/ipa/ipa_modem.c
index a6f6cd149c1b6..c7a0b167c4326 100644
--- a/drivers/net/ipa/ipa_modem.c
+++ b/drivers/net/ipa/ipa_modem.c
@@ -110,13 +110,16 @@ static int ipa_stop(struct net_device *netdev)
 	return 0;
 }
 
-/** ipa_start_xmit() - Transmits an skb.
- * @skb: skb to be transmitted
- * @dev: network device
+/** ipa_start_xmit() - Transmit an skb
+ * @skb:	Socket buffer to be transmitted
+ * @netdev:	Network device
  *
- * Return codes:
- * NETDEV_TX_OK: Success
- * NETDEV_TX_BUSY: Error while transmitting the skb. Try again later
+ * Return: NETDEV_TX_OK if successful (or dropped), NETDEV_TX_BUSY otherwise
+
+ * Normally NETDEV_TX_OK indicates the buffer was successfully transmitted.
+ * If the buffer has an unexpected protocol or its size is out of range it
+ * is quietly dropped, returning NETDEV_TX_OK.  NETDEV_TX_BUSY indicates
+ * the buffer cannot be sent at this time and should retried later.
  */
 static netdev_tx_t
 ipa_start_xmit(struct sk_buff *skb, struct net_device *netdev)
@@ -136,7 +139,25 @@ ipa_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 	if (endpoint->config.qmap && skb->protocol != htons(ETH_P_MAP))
 		goto err_drop_skb;
 
-	/* The hardware must be powered for us to transmit */
+	/* The hardware must be powered for us to transmit, so if we're not
+	 * ready we want the network stack to stop queueing until power is
+	 * ACTIVE.  Once runtime resume has completed, we inform the network
+	 * stack it's OK to try transmitting again.
+	 *
+	 * We learn from pm_runtime_get() whether the hardware is powered.
+	 * If it was not, powering up is either started or already underway.
+	 * And in that case we want to disable queueing, expecting it to be
+	 * re-enabled once power is ACTIVE.  But runtime PM and network
+	 * transmit run concurrently, and if we're not careful the requests
+	 * to stop and start queueing could occur in the wrong order.
+	 *
+	 * For that reason we *always* stop queueing here, *before* the call
+	 * to pm_runtime_get().  If we determine here that power is ACTIVE,
+	 * we restart queueing before transmitting the SKB.  Otherwise
+	 * queueing will eventually be enabled after resume completes.
+	 */
+	ipa_power_modem_queue_stop(ipa);
+
 	dev = &ipa->pdev->dev;
 	ret = pm_runtime_get(dev);
 	if (ret < 1) {
@@ -147,12 +168,6 @@ ipa_start_xmit(struct sk_buff *skb, struct net_device *netdev)
 			goto err_drop_skb;
 		}
 
-		/* No power (yet).  Stop the network stack from transmitting
-		 * until we're resumed; ipa_modem_resume() arranges for the
-		 * TX queue to be started again.
-		 */
-		ipa_power_modem_queue_stop(ipa);
-
 		pm_runtime_put_noidle(dev);
 
 		return NETDEV_TX_BUSY;
diff --git a/drivers/net/ipa/ipa_power.c b/drivers/net/ipa/ipa_power.c
index e223886123cea..f1802448ff447 100644
--- a/drivers/net/ipa/ipa_power.c
+++ b/drivers/net/ipa/ipa_power.c
@@ -233,28 +233,32 @@ void ipa_power_suspend_handler(struct ipa *ipa, enum ipa_irq_id irq_id)
 	ipa_interrupt_suspend_clear_all(ipa->interrupt);
 }
 
-/* The next few functions coordinate stopping and starting the modem
+/* The next few functions are used when stopping and starting the modem
  * network device transmit queue.
  *
- * Transmit can be running concurrent with power resume, and there's a
- * chance the resume completes before the transmit path stops the queue,
- * leaving the queue in a stopped state.  The next two functions are used
- * to avoid this: ipa_power_modem_queue_stop() is used by ipa_start_xmit()
- * to conditionally stop the TX queue; and ipa_power_modem_queue_start()
- * is used by ipa_runtime_resume() to conditionally restart it.
+ * Transmit can run concurrent with power resume.  When transmitting,
+ * we disable further transmits until we can determine whether power
+ * is ACTIVE.  If it is, future transmits are re-enabled and the buffer
+ * gets sent (or dropped).  If power is not ACTIVE, it will eventually
+ * be, and transmits stay disabled until after it is.
  *
- * Two flags and a spinlock are used.  If the queue is stopped, the STOPPED
- * power flag is set.  And if the queue is started, the STARTED flag is set.
- * The queue is only started on resume if the STOPPED flag is set.  And the
- * queue is only started in ipa_start_xmit() if the STARTED flag is *not*
- * set.  As a result, the queue remains operational if the two activites
- * happen concurrently regardless of the order they complete.  The spinlock
- * ensures the flag and TX queue operations are done atomically.
+ * Two flags and a spinlock are used when managing this.  If the queue
+ * is stopped, the STOPPED power flag is set.  And if the queue is
+ * started, the STARTED flag is set.
  *
  * The first function stops the modem netdev transmit queue, but only if
- * the STARTED flag is *not* set.  That flag is cleared if it was set.
- * If the queue is stopped, the STOPPED flag is set.  This is called only
- * from the power ->runtime_resume operation.
+ * the STARTED flag is *not* set.  This previously avoided a race where
+ * the TX path stops further transmits after power has become ACTIVE.
+ * The STARTED flag is cleared by this function.
+ *
+ * The second function starts the transmit queue, but only if the
+ * STOPPED flag is set.  This avoids enabling transmits repeatedly
+ * immediately after power has become ACTIVE (not really a big deal).
+ * If the STOPPED flag was set, it is cleared and the STARTED flag
+ * is set by this function.
+ *
+ * The third function enables transmits again and clears the STARTED
+ * flag in case it was set, to return it to initial state.
  */
 void ipa_power_modem_queue_stop(struct ipa *ipa)
 {
@@ -291,9 +295,13 @@ void ipa_power_modem_queue_wake(struct ipa *ipa)
 	spin_unlock_irqrestore(&power->spinlock, flags);
 }
 
-/* This function clears the STARTED flag once the TX queue is operating */
+/* This function is run after power has become ACTIVE.  It enables transmits
+ * again clears the STARTED flag to indicate the TX queue is operating and
+ * can be stopped again if necessary.
+ */
 void ipa_power_modem_queue_active(struct ipa *ipa)
 {
+	netif_wake_queue(ipa->modem_netdev);
 	clear_bit(IPA_POWER_FLAG_STARTED, ipa->power->flags);
 }
 
-- 
2.40.1