2010-11-29 14:54:47

by Johannes Stezenbach

[permalink] [raw]
Subject: [PATCH RFC] mac80211: fix "NOHZ: local_softirq_pending 08"

ieee80211_tx_status() documentation says "This function may not be
called in IRQ context", and it is called by rt2800usb
from a workqueue context. Thus it needs to call
netif_rx_ni() instead of netif_rx().
This change fixes the "NOHZ: local_softirq_pending 08"
messages I've been getting with rt2800usb.

Signed-off-by: Johannes Stezenbach <[email protected]>
--
net/mac80211/status.c | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/net/mac80211/status.c b/net/mac80211/status.c
index bed7e32..ba8dd8e 100644
--- a/net/mac80211/status.c
+++ b/net/mac80211/status.c
@@ -403,7 +403,7 @@ void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
skb2 = skb_clone(skb, GFP_ATOMIC);
if (skb2) {
skb2->dev = prev_dev;
- netif_rx(skb2);
+ netif_rx_ni(skb2);
}
}

@@ -412,7 +412,7 @@ void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
}
if (prev_dev) {
skb->dev = prev_dev;
- netif_rx(skb);
+ netif_rx_ni(skb);
skb = NULL;
}
rcu_read_unlock();


2010-11-29 15:37:38

by Johannes Berg

[permalink] [raw]
Subject: Re: [PATCH RFC] mac80211: fix "NOHZ: local_softirq_pending 08"

On Mon, 2010-11-29 at 16:27 +0100, Johannes Stezenbach wrote:

> > That's kinda pointless though for drivers that already call it from a
> > tasklet or similar -- how about instead adding an
> > ieee80211_tx_status_ni() inline along the lines of ieee80211_rx_ni()?
>
> It gets confusing...
> There already is ieee80211_tx_status_irqsafe(), but you want a third
> option, right?

Yes: _irqsafe has to go through the tasklet, _ni just has to disable
local BHs.

johannes


2010-11-29 16:13:25

by Johannes Stezenbach

[permalink] [raw]
Subject: [PATCH v3 RFC] mac80211: add ieee80211_tx_status_ni()

ieee80211_tx_status() documentation says "This function may not be
called in IRQ context", and it is called by rt2800usb
from a workqueue context. However, ieee80211_tx_status() is
meant to be called from tasklets and thus uses netif_rx().
Add a new ieee80211_tx_status_ni() which does the same
thing but is safe to be called from process context.

Using ieee80211_tx_status_ni() for rt2800usb fixes the
"NOHZ: local_softirq_pending 08" messages I've been getting.

Signed-off-by: Johannes Stezenbach <[email protected]>
---
v2: add ieee80211_tx_status_ni()
v3: simplify accorindg to review

Note: the patch is now incomplete, rt2x00lib_txdone() is used by
several drivers, rt2800usb calls it from a workqueue while
e.g. while rt2800pci calls it from a tasklet
(rt2800pci_txstatus_tasklet). I need to sort this out with
the rt2x00 developers.


include/net/mac80211.h | 28 ++++++++++++++++++++++++----
1 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/include/net/mac80211.h b/include/net/mac80211.h
index eaa4aff..e411cf8 100644
--- a/include/net/mac80211.h
+++ b/include/net/mac80211.h
@@ -2055,8 +2055,8 @@ static inline void ieee80211_rx_ni(struct ieee80211_hw *hw,
*
* This function may not be called in IRQ context. Calls to this function
* for a single hardware must be synchronized against each other. Calls
- * to this function and ieee80211_tx_status_irqsafe() may not be mixed
- * for a single hardware.
+ * to this function, ieee80211_tx_status_ni() and ieee80211_tx_status_irqsafe()
+ * may not be mixed for a single hardware.
*
* @hw: the hardware the frame was transmitted by
* @skb: the frame that was transmitted, owned by mac80211 after this call
@@ -2065,13 +2065,33 @@ void ieee80211_tx_status(struct ieee80211_hw *hw,
struct sk_buff *skb);

/**
+ * ieee80211_tx_status_ni - transmit status callback (in process context)
+ *
+ * Like ieee80211_tx_status() but can be called in process context.
+ *
+ * Calls to this function, ieee80211_tx_status() and
+ * ieee80211_tx_status_irqsafe() may not be mixed
+ * for a single hardware.
+ *
+ * @hw: the hardware the frame was transmitted by
+ * @skb: the frame that was transmitted, owned by mac80211 after this call
+ */
+static inline void ieee80211_tx_status_ni(struct ieee80211_hw *hw,
+ struct sk_buff *skb)
+{
+ local_bh_disable();
+ ieee80211_tx_status(hw, skb);
+ local_bh_enable();
+}
+
+/**
* ieee80211_tx_status_irqsafe - IRQ-safe transmit status callback
*
* Like ieee80211_tx_status() but can be called in IRQ context
* (internally defers to a tasklet.)
*
- * Calls to this function and ieee80211_tx_status() may not be mixed for a
- * single hardware.
+ * Calls to this function, ieee80211_tx_status() and
+ * ieee80211_tx_status_ni() may not be mixed for a single hardware.
*
* @hw: the hardware the frame was transmitted by
* @skb: the frame that was transmitted, owned by mac80211 after this call

2010-11-29 15:27:44

by Johannes Stezenbach

[permalink] [raw]
Subject: Re: [PATCH RFC] mac80211: fix "NOHZ: local_softirq_pending 08"

On Mon, Nov 29, 2010 at 04:09:09PM +0100, Johannes Berg wrote:
> On Mon, 2010-11-29 at 15:54 +0100, Johannes Stezenbach wrote:
> > ieee80211_tx_status() documentation says "This function may not be
> > called in IRQ context", and it is called by rt2800usb
> > from a workqueue context. Thus it needs to call
> > netif_rx_ni() instead of netif_rx().
> > This change fixes the "NOHZ: local_softirq_pending 08"
> > messages I've been getting with rt2800usb.
>
> > - netif_rx(skb2);
> > + netif_rx_ni(skb2);
>
> That's kinda pointless though for drivers that already call it from a
> tasklet or similar -- how about instead adding an
> ieee80211_tx_status_ni() inline along the lines of ieee80211_rx_ni()?

It gets confusing...
There already is ieee80211_tx_status_irqsafe(), but you want a third
option, right?

Johannes

2010-11-30 15:53:02

by Johannes Berg

[permalink] [raw]
Subject: Re: [PATCH v4] mac80211/rt2x00: add ieee80211_tx_status_ni()

On Tue, 2010-11-30 at 16:49 +0100, Johannes Stezenbach wrote:

> * This function may not be called in IRQ context. Calls to this function
> * for a single hardware must be synchronized against each other. Calls
> - * to this function and ieee80211_tx_status_irqsafe() may not be mixed
> - * for a single hardware.
> + * to this function, ieee80211_tx_status_ni() and ieee80211_tx_status_irqsafe()
> + * may not be mixed for a single hardware.

I'm ok with this, although technically you can mix
ieee80211_tx_status_ni() and ieee80211_tx_status(), just not either or
both of them with _irqsafe().

johannes


2010-11-29 15:09:12

by Johannes Berg

[permalink] [raw]
Subject: Re: [PATCH RFC] mac80211: fix "NOHZ: local_softirq_pending 08"

On Mon, 2010-11-29 at 15:54 +0100, Johannes Stezenbach wrote:
> ieee80211_tx_status() documentation says "This function may not be
> called in IRQ context", and it is called by rt2800usb
> from a workqueue context. Thus it needs to call
> netif_rx_ni() instead of netif_rx().
> This change fixes the "NOHZ: local_softirq_pending 08"
> messages I've been getting with rt2800usb.

> - netif_rx(skb2);
> + netif_rx_ni(skb2);

That's kinda pointless though for drivers that already call it from a
tasklet or similar -- how about instead adding an
ieee80211_tx_status_ni() inline along the lines of ieee80211_rx_ni()?

johannes


2010-11-30 16:35:05

by Johannes Stezenbach

[permalink] [raw]
Subject: Re: [PATCH v4] mac80211/rt2x00: add ieee80211_tx_status_ni()

On Tue, Nov 30, 2010 at 04:53:00PM +0100, Johannes Berg wrote:
> On Tue, 2010-11-30 at 16:49 +0100, Johannes Stezenbach wrote:
>
> > * This function may not be called in IRQ context. Calls to this function
> > * for a single hardware must be synchronized against each other. Calls
> > - * to this function and ieee80211_tx_status_irqsafe() may not be mixed
> > - * for a single hardware.
> > + * to this function, ieee80211_tx_status_ni() and ieee80211_tx_status_irqsafe()
> > + * may not be mixed for a single hardware.
>
> I'm ok with this, although technically you can mix
> ieee80211_tx_status_ni() and ieee80211_tx_status(), just not either or
> both of them with _irqsafe().

I copied these from ieee80211_rx() etc. since I wasn't sure if there's
not some subtlety I didn't get. One can mix ieee80211_rx() and
ieee80211_rx_ni(), too, right?


Thanks
Johannes

2010-11-29 15:51:34

by Johannes Stezenbach

[permalink] [raw]
Subject: [PATCH v2 RFC] mac80211: fix "NOHZ: local_softirq_pending 08"

ieee80211_tx_status() documentation says "This function may not be
called in IRQ context", and it is called by rt2800usb
from a workqueue context. However, ieee80211_tx_status() is
meant to be called from tasklets and thus uses netif_rx().
Add a new ieee80211_tx_status_ni() which does the same
thing but uses netif_rx_ni() instead of netif_rx().

This change fixes the "NOHZ: local_softirq_pending 08"
messages I've been getting with rt2800usb.

Signed-off-by: Johannes Stezenbach <[email protected]>
---
v2: add ieee80211_tx_status_ni()

Note: the patch is now incomplete, rt2x00lib_txdone() is used by
several drivers and I'm currently checking if ieee80211_tx_status_ni()
can be used by all of them of if I need to split rt2x00lib_txdone()
in two versions. An updated patch with the rt2x00
changes will follow, but I'm seeking feedback if the base
change here is OK.


include/net/mac80211.h | 23 +++++++++++++++++++----
net/mac80211/status.c | 19 ++++++++++++++++---
2 files changed, 35 insertions(+), 7 deletions(-)

diff --git a/include/net/mac80211.h b/include/net/mac80211.h
index eaa4aff..df42312 100644
--- a/include/net/mac80211.h
+++ b/include/net/mac80211.h
@@ -2055,8 +2055,8 @@ static inline void ieee80211_rx_ni(struct ieee80211_hw *hw,
*
* This function may not be called in IRQ context. Calls to this function
* for a single hardware must be synchronized against each other. Calls
- * to this function and ieee80211_tx_status_irqsafe() may not be mixed
- * for a single hardware.
+ * to this function, ieee80211_tx_status_ni() and ieee80211_tx_status_irqsafe()
+ * may not be mixed for a single hardware.
*
* @hw: the hardware the frame was transmitted by
* @skb: the frame that was transmitted, owned by mac80211 after this call
@@ -2065,13 +2065,28 @@ void ieee80211_tx_status(struct ieee80211_hw *hw,
struct sk_buff *skb);

/**
+ * ieee80211_tx_status_ni - transmit status callback (in process context)
+ *
+ * Like ieee80211_tx_status() but can be called in process context.
+ *
+ * Calls to this function, ieee80211_tx_status() and
+ * ieee80211_tx_status_irqsafe() may not be mixed
+ * for a single hardware.
+ *
+ * @hw: the hardware the frame was transmitted by
+ * @skb: the frame that was transmitted, owned by mac80211 after this call
+ */
+void ieee80211_tx_status_ni(struct ieee80211_hw *hw,
+ struct sk_buff *skb);
+
+/**
* ieee80211_tx_status_irqsafe - IRQ-safe transmit status callback
*
* Like ieee80211_tx_status() but can be called in IRQ context
* (internally defers to a tasklet.)
*
- * Calls to this function and ieee80211_tx_status() may not be mixed for a
- * single hardware.
+ * Calls to this function, ieee80211_tx_status() and
+ * ieee80211_tx_status_ni() may not be mixed for a single hardware.
*
* @hw: the hardware the frame was transmitted by
* @skb: the frame that was transmitted, owned by mac80211 after this call
diff --git a/net/mac80211/status.c b/net/mac80211/status.c
index bed7e32..ce4d239 100644
--- a/net/mac80211/status.c
+++ b/net/mac80211/status.c
@@ -170,7 +170,8 @@ static void ieee80211_frame_acked(struct sta_info *sta, struct sk_buff *skb)
*/
#define STA_LOST_PKT_THRESHOLD 50

-void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
+void __ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb,
+ int (*netif_rx_func)(struct sk_buff *skb))
{
struct sk_buff *skb2;
struct ieee80211_hdr *hdr = (struct ieee80211_hdr *) skb->data;
@@ -403,7 +404,7 @@ void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
skb2 = skb_clone(skb, GFP_ATOMIC);
if (skb2) {
skb2->dev = prev_dev;
- netif_rx(skb2);
+ netif_rx_func(skb2);
}
}

@@ -412,10 +413,22 @@ void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
}
if (prev_dev) {
skb->dev = prev_dev;
- netif_rx(skb);
+ netif_rx_func(skb);
skb = NULL;
}
rcu_read_unlock();
dev_kfree_skb(skb);
}
+
+void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
+{
+ __ieee80211_tx_status(hw, skb, netif_rx);
+}
EXPORT_SYMBOL(ieee80211_tx_status);
+
+void ieee80211_tx_status_ni(struct ieee80211_hw *hw,
+ struct sk_buff *skb)
+{
+ __ieee80211_tx_status(hw, skb, netif_rx_ni);
+}
+EXPORT_SYMBOL(ieee80211_tx_status_ni);

2010-11-30 18:44:42

by John W. Linville

[permalink] [raw]
Subject: Re: [PATCH v4] mac80211/rt2x00: add ieee80211_tx_status_ni()

On Tue, Nov 30, 2010 at 05:35:58PM +0100, Johannes Berg wrote:
> On Tue, 2010-11-30 at 17:34 +0100, Johannes Stezenbach wrote:
> > On Tue, Nov 30, 2010 at 04:53:00PM +0100, Johannes Berg wrote:
> > > On Tue, 2010-11-30 at 16:49 +0100, Johannes Stezenbach wrote:
> > >
> > > > * This function may not be called in IRQ context. Calls to this function
> > > > * for a single hardware must be synchronized against each other. Calls
> > > > - * to this function and ieee80211_tx_status_irqsafe() may not be mixed
> > > > - * for a single hardware.
> > > > + * to this function, ieee80211_tx_status_ni() and ieee80211_tx_status_irqsafe()
> > > > + * may not be mixed for a single hardware.
> > >
> > > I'm ok with this, although technically you can mix
> > > ieee80211_tx_status_ni() and ieee80211_tx_status(), just not either or
> > > both of them with _irqsafe().
> >
> > I copied these from ieee80211_rx() etc. since I wasn't sure if there's
> > not some subtlety I didn't get. One can mix ieee80211_rx() and
> > ieee80211_rx_ni(), too, right?
>
> Yes. Note that all this doesn't make any sense though, since you must
> never call them concurrently. And it's hard to imagine a situation where
> a single driver calls _ni and non-_ni versions, while making sure they
> can't both happen at the same time ...

Maybe someone can clarify the wording of the comments and send a follow-on patch?

Thanks,

John
--
John W. Linville Someday the world will need a hero, and you
[email protected] might be all we have. Be ready.

2010-11-30 15:49:32

by Johannes Stezenbach

[permalink] [raw]
Subject: [PATCH v4] mac80211/rt2x00: add ieee80211_tx_status_ni()

All rt2x00 drivers except rt2800pci call ieee80211_tx_status() from
a workqueue, which causes "NOHZ: local_softirq_pending 08" messages.

To fix it, add ieee80211_tx_status_ni() similar to ieee80211_rx_ni()
which can be called from process context, and call it from
rt2x00lib_txdone(). For the rt2800pci special case a driver
flag is introduced.

Signed-off-by: Johannes Stezenbach <[email protected]>
---
v2: add ieee80211_tx_status_ni()
v3: simplify accorindg to review
v4: add rt2x00 as first user of ieee80211_tx_status_ni()

The DRIVER_REQUIRE_TASKLET_CONTEXT flag may seem ugly, but there is
a pending rewrite of the rt2800pci irq handling code by
Helmut, so we decided to use this easy solution.

Note: Only tested with rt2800usb.

drivers/net/wireless/rt2x00/rt2800pci.c | 1 +
drivers/net/wireless/rt2x00/rt2x00.h | 1 +
drivers/net/wireless/rt2x00/rt2x00dev.c | 9 ++++++---
include/net/mac80211.h | 28 ++++++++++++++++++++++++----
4 files changed, 32 insertions(+), 7 deletions(-)

diff --git a/drivers/net/wireless/rt2x00/rt2800pci.c b/drivers/net/wireless/rt2x00/rt2800pci.c
index b48d2d3..3ef0228 100644
--- a/drivers/net/wireless/rt2x00/rt2800pci.c
+++ b/drivers/net/wireless/rt2x00/rt2800pci.c
@@ -932,6 +932,7 @@ static int rt2800pci_probe_hw(struct rt2x00_dev *rt2x00dev)
__set_bit(DRIVER_REQUIRE_DMA, &rt2x00dev->flags);
__set_bit(DRIVER_REQUIRE_L2PAD, &rt2x00dev->flags);
__set_bit(DRIVER_REQUIRE_TXSTATUS_FIFO, &rt2x00dev->flags);
+ __set_bit(DRIVER_REQUIRE_TASKLET_CONTEXT, &rt2x00dev->flags);
if (!modparam_nohwcrypt)
__set_bit(CONFIG_SUPPORT_HW_CRYPTO, &rt2x00dev->flags);
__set_bit(DRIVER_SUPPORT_LINK_TUNING, &rt2x00dev->flags);
diff --git a/drivers/net/wireless/rt2x00/rt2x00.h b/drivers/net/wireless/rt2x00/rt2x00.h
index 556b744..28ea59a 100644
--- a/drivers/net/wireless/rt2x00/rt2x00.h
+++ b/drivers/net/wireless/rt2x00/rt2x00.h
@@ -670,6 +670,7 @@ enum rt2x00_flags {
DRIVER_REQUIRE_COPY_IV,
DRIVER_REQUIRE_L2PAD,
DRIVER_REQUIRE_TXSTATUS_FIFO,
+ DRIVER_REQUIRE_TASKLET_CONTEXT,

/*
* Driver features
diff --git a/drivers/net/wireless/rt2x00/rt2x00dev.c b/drivers/net/wireless/rt2x00/rt2x00dev.c
index 87a421b..fa74acd 100644
--- a/drivers/net/wireless/rt2x00/rt2x00dev.c
+++ b/drivers/net/wireless/rt2x00/rt2x00dev.c
@@ -376,9 +376,12 @@ void rt2x00lib_txdone(struct queue_entry *entry,
* through a mac80211 library call (RTS/CTS) then we should not
* send the status report back.
*/
- if (!(skbdesc_flags & SKBDESC_NOT_MAC80211))
- ieee80211_tx_status(rt2x00dev->hw, entry->skb);
- else
+ if (!(skbdesc_flags & SKBDESC_NOT_MAC80211)) {
+ if (test_bit(DRIVER_REQUIRE_TASKLET_CONTEXT, &rt2x00dev->flags))
+ ieee80211_tx_status(rt2x00dev->hw, entry->skb);
+ else
+ ieee80211_tx_status_ni(rt2x00dev->hw, entry->skb);
+ } else
dev_kfree_skb_any(entry->skb);

/*
diff --git a/include/net/mac80211.h b/include/net/mac80211.h
index eaa4aff..e411cf8 100644
--- a/include/net/mac80211.h
+++ b/include/net/mac80211.h
@@ -2055,8 +2055,8 @@ static inline void ieee80211_rx_ni(struct ieee80211_hw *hw,
*
* This function may not be called in IRQ context. Calls to this function
* for a single hardware must be synchronized against each other. Calls
- * to this function and ieee80211_tx_status_irqsafe() may not be mixed
- * for a single hardware.
+ * to this function, ieee80211_tx_status_ni() and ieee80211_tx_status_irqsafe()
+ * may not be mixed for a single hardware.
*
* @hw: the hardware the frame was transmitted by
* @skb: the frame that was transmitted, owned by mac80211 after this call
@@ -2065,13 +2065,33 @@ void ieee80211_tx_status(struct ieee80211_hw *hw,
struct sk_buff *skb);

/**
+ * ieee80211_tx_status_ni - transmit status callback (in process context)
+ *
+ * Like ieee80211_tx_status() but can be called in process context.
+ *
+ * Calls to this function, ieee80211_tx_status() and
+ * ieee80211_tx_status_irqsafe() may not be mixed
+ * for a single hardware.
+ *
+ * @hw: the hardware the frame was transmitted by
+ * @skb: the frame that was transmitted, owned by mac80211 after this call
+ */
+static inline void ieee80211_tx_status_ni(struct ieee80211_hw *hw,
+ struct sk_buff *skb)
+{
+ local_bh_disable();
+ ieee80211_tx_status(hw, skb);
+ local_bh_enable();
+}
+
+/**
* ieee80211_tx_status_irqsafe - IRQ-safe transmit status callback
*
* Like ieee80211_tx_status() but can be called in IRQ context
* (internally defers to a tasklet.)
*
- * Calls to this function and ieee80211_tx_status() may not be mixed for a
- * single hardware.
+ * Calls to this function, ieee80211_tx_status() and
+ * ieee80211_tx_status_ni() may not be mixed for a single hardware.
*
* @hw: the hardware the frame was transmitted by
* @skb: the frame that was transmitted, owned by mac80211 after this call
--
1.7.2.3


2010-11-29 15:58:42

by Johannes Berg

[permalink] [raw]
Subject: Re: [PATCH v2 RFC] mac80211: fix "NOHZ: local_softirq_pending 08"

On Mon, 2010-11-29 at 16:51 +0100, Johannes Stezenbach wrote:


> -void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
> +void __ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb,
> + int (*netif_rx_func)(struct sk_buff *skb))

This seems weird to me -- why not just do

static inline void ieee80211_tx_status_ni(...)
{
local_bh_disable();
ieee80211_tx_status(...);
local_bh_enable();
}

johannes


2010-11-30 16:35:59

by Johannes Berg

[permalink] [raw]
Subject: Re: [PATCH v4] mac80211/rt2x00: add ieee80211_tx_status_ni()

On Tue, 2010-11-30 at 17:34 +0100, Johannes Stezenbach wrote:
> On Tue, Nov 30, 2010 at 04:53:00PM +0100, Johannes Berg wrote:
> > On Tue, 2010-11-30 at 16:49 +0100, Johannes Stezenbach wrote:
> >
> > > * This function may not be called in IRQ context. Calls to this function
> > > * for a single hardware must be synchronized against each other. Calls
> > > - * to this function and ieee80211_tx_status_irqsafe() may not be mixed
> > > - * for a single hardware.
> > > + * to this function, ieee80211_tx_status_ni() and ieee80211_tx_status_irqsafe()
> > > + * may not be mixed for a single hardware.
> >
> > I'm ok with this, although technically you can mix
> > ieee80211_tx_status_ni() and ieee80211_tx_status(), just not either or
> > both of them with _irqsafe().
>
> I copied these from ieee80211_rx() etc. since I wasn't sure if there's
> not some subtlety I didn't get. One can mix ieee80211_rx() and
> ieee80211_rx_ni(), too, right?

Yes. Note that all this doesn't make any sense though, since you must
never call them concurrently. And it's hard to imagine a situation where
a single driver calls _ni and non-_ni versions, while making sure they
can't both happen at the same time ...

johannes