Return-path: Received: from flock1.newmail.ru ([82.204.219.207]:57264 "HELO flock1.newmail.ru" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1751454AbZAFJwA (ORCPT ); Tue, 6 Jan 2009 04:52:00 -0500 From: Andrey Borzenkov To: Dave Subject: Re: 2.6.28: warn_slowpath in orinoco receive path Date: Tue, 6 Jan 2009 12:51:39 +0300 Cc: orinoco-devel@lists.sourceforge.net, linux-wireless@vger.kernel.org, linux-kernel@vger.kernel.org References: <200901052005.12428.arvidjaar@mail.ru> <49626E4D.3010705@gmail.com> In-Reply-To: <49626E4D.3010705@gmail.com> MIME-Version: 1.0 Content-Type: Multipart/Mixed; boundary="Boundary-00=_tmyYJ93IgGZkXNg" Message-Id: <200901061251.41628.arvidjaar@mail.ru> (sfid-20090106_105211_232148_80240BA1) Sender: linux-wireless-owner@vger.kernel.org List-ID: --Boundary-00=_tmyYJ93IgGZkXNg Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On =D0=9F=D0=BE=D0=BD=D0=B5=D0=B4=D0=B5=D0=BB=D1=8C=D0=BD=D0=B8=D0=BA 05 = =D1=8F=D0=BD=D0=B2=D0=B0=D1=80=D1=8F 2009 23:32:13 Dave wrote: > Looks like the RX interrupt occurred at an inconvenient point during > the list_del call in the RX tasklet (orinoco_rx_isr_tasklet). > > The call needs to be protected from the RX interrupt. > > Quick patch included below - I'm not sure that the local_irq_* > functions are the ones we need, but it compiles and runs. > As was already pointed out, we can't be sure tasklet runs on the same=20 CPU as interrupt handler. What about attached patch? It actually moves=20 list to temporary head which can be processed without races; the idea is=20 to minimize amount and number of times we need to disable interrupts.=20 Patch compiles and runs :) > I don't suppose you're able to reproduce the error? > Right. By the way. Agere driver takes different approach. The only thing it=20 does in interrupt handler directly is to turn off Hermes interrupts and=20 start off another thread to process pending events. After all events are=20 processed interrupts are enabled again. It means the bulk of code is=20 executed in non-interrupt context; and looking how much is done in=20 orinoco driver during interrupt processing, this does not sound like bad=20 idea. Do you see any obvious cons here? --Boundary-00=_tmyYJ93IgGZkXNg Content-Type: message/rfc822; name="orinoco-rx_list-protect" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="orinoco-rx_list-protect" Subject: [PATCH] orinoco: protect rx_list manipulation in orinoco_rx_isr_tasklet From: Andrey Borzenkov The list is changed both in normal and interrupt context. Protect manipulation in non-interrupt case; this hopefully fixes this warning: [89479.105038] WARNING: at /home/bor/src/linux-git/lib/list_debug.c:30 __list_add+0x8f/0xa0() [89479.105058] list_add corruption. prev->next should be next (dddb3568), but was cbc28978. (prev=dddb3568). [89479.106002] Pid: 15746, comm: X Not tainted 2.6.28-1avb #26 [89479.106020] Call Trace: [89479.106062] [] warn_slowpath+0x60/0x80 [89479.106104] [] ? native_sched_clock+0x20/0x70 [89479.106194] [] ? lock_release_holdtime+0x35/0x200 [89479.106218] [] ? __slab_alloc+0x550/0x560 [89479.106254] [] ? _spin_unlock+0x1d/0x20 [89479.106270] [] ? __slab_alloc+0x550/0x560 [89479.106302] [] ? delay_tsc+0x17/0x24 [89479.106319] [] ? __const_udelay+0x21/0x30 [89479.106376] [] ? hermes_bap_seek+0x112/0x1e0 [hermes] [89479.106396] [] ? trace_hardirqs_off+0xb/0x10 [89479.106418] [] ? __kmalloc_track_caller+0xb7/0x110 [89479.106448] [] ? dev_alloc_skb+0x1c/0x30 [89479.106465] [] ? dev_alloc_skb+0x1c/0x30 [89479.106482] [] __list_add+0x8f/0xa0 [89479.106551] [] orinoco_interrupt+0xcae/0x16c0 [orinoco] [89479.106574] [] ? tick_dev_program_event+0x33/0xb0 [89479.106594] [] ? native_sched_clock+0x20/0x70 [89479.106613] [] ? lock_release_holdtime+0x35/0x200 [89479.106662] [] ? trace_hardirqs_off+0xb/0x10 [89479.106892] [] ? usb_hcd_irq+0x97/0xa0 [usbcore] [89479.106926] [] handle_IRQ_event+0x29/0x60 [89479.106947] [] handle_level_irq+0x69/0xe0 [89479.106963] [] ? handle_level_irq+0x0/0xe0 [89479.106977] [] ? tcp_v4_rcv+0x633/0x6e0 [89479.107025] [] ? common_interrupt+0x28/0x30 [89479.107057] [] ? sk_run_filter+0x320/0x7a0 [89479.107078] [] ? list_del+0x21/0x90 [89479.107106] [] ? orinoco_rx_isr_tasklet+0x2ce/0x480 [orinoco] [89479.107131] [] ? __lock_acquire+0x160/0x1650 [89479.107151] [] ? native_sched_clock+0x20/0x70 [89479.107169] [] ? lock_release_holdtime+0x35/0x200 [89479.107200] [] ? irq_enter+0xa/0x60 [89479.107217] [] ? do_IRQ+0xd2/0x130 [89479.107518] [] ? restore_nocheck_notrace+0x0/0xe [89479.107542] [] ? __do_softirq+0x0/0x110 [89479.107561] [] ? trace_hardirqs_on_caller+0x74/0x140 [89479.107583] [] ? trace_hardirqs_on_thunk+0xc/0x10 [89479.107602] [] ? tasklet_action+0x27/0x90 [89479.107620] [] ? trace_hardirqs_on_caller+0x74/0x140 [89479.107638] [] ? tasklet_action+0x43/0x90 [89479.107655] [] ? __do_softirq+0x6f/0x110 [89479.107674] [] ? __do_softirq+0x0/0x110 [89479.107685] [] ? handle_level_irq+0x0/0xe0 [89479.107715] [] ? irq_exit+0x5d/0x80 [89479.107732] [] ? do_IRQ+0xd2/0x130 [89479.107747] [] ? sysenter_exit+0xf/0x16 [89479.107765] [] ? trace_hardirqs_on_caller+0xfd/0x140 [89479.107782] [] ? common_interrupt+0x28/0x30 [89479.107797] ---[ end trace a1fc0a52df4a729d ]--- Patch based on suggestion of Dave but using spinlock instead of local_irq_*. Signed-off-by: Andrey Borzenkov --- drivers/net/wireless/orinoco.c | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-) diff --git a/drivers/net/wireless/orinoco.c b/drivers/net/wireless/orinoco.c index 1d7e14b..09fc080 100644 --- a/drivers/net/wireless/orinoco.c +++ b/drivers/net/wireless/orinoco.c @@ -1568,9 +1568,15 @@ static void orinoco_rx_isr_tasklet(unsigned long data) struct orinoco_rx_data *rx_data, *temp; struct hermes_rx_descriptor *desc; struct sk_buff *skb; + struct list_head temp_rx_list; + + /* Move list to temporary head to avoid races with interrupt handler */ + spin_lock_irq(&priv->lock); + list_replace_init(&priv->rx_list, &temp_rx_list); + spin_unlock_irq(&priv->lock); /* extract desc and skb from queue */ - list_for_each_entry_safe(rx_data, temp, &priv->rx_list, list) { + list_for_each_entry_safe(rx_data, temp, &temp_rx_list, list) { desc = rx_data->desc; skb = rx_data->skb; list_del(&rx_data->list); --Boundary-00=_tmyYJ93IgGZkXNg--