We often see crashes in work-queue processing when deleting
lots of wifi station interfaces. I'm guessing that there is probably
a work item that was not properly un-registered before deleting
memory. I have backported some wifi fixes from upstream, so
maybe they are to blame, but in case anyone has any suggestions
for places to look, please let me know.
For reference, my tree is here:
http://dmz2.candelatech.com/git/gitweb.cgi?p=linux-3.7.dev.y/.git;a=summary
I'll go poke at the code in the meantime.
wiphy1: start_sw_scan: running-other-vifs: 0 running-station-vifs: 159, associated-stations: 156 scanning current channel: 5745 MHz
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff81099d58>] cwq_dec_nr_in_flight+0x46/0xd5
sta110: deauthenticating from 30:46:9a:10:0b:9c by local choice (reason=3)
PGD 0
Oops: 0000 [#1] PREEMPT SMP
Modules linked in: nfsv3 nfs_acl nfnetlink_log nfnetlink bluetooth nfsv4 auth_rpcgss nfs fscache nf_nat_ipv4 nf_nat fuse 8021q garp stp llc macvlan lockd
wanlink(O) sunrpc pktgen f71882fg coretemp hwmon mperf iTCO_wdt iTCO_vendor_support kvm cdc_acm ppdev gpio_ich snd_hda_codec_realtek i2c_i801 microcode
serio_raw lpc_ich pcspkr ath9k ath9k_common ath9k_hw ath mac80211 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e snd_page_alloc
snd_timer snd soundcore cfg80211 parport_pc parport uinput ipv6 i915 video i2c_algo_bit drm_kms_helper drm i2c_core [last unloaded: iptable_nat]
CPU 1
Pid: 15954, comm: kworker/u:2 Tainted: G WC O 3.7.6+ #65 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff81099d58>] [<ffffffff81099d58>] cwq_dec_nr_in_flight+0x46/0xd5
RSP: 0018:ffff8800c6ed9dc8 EFLAGS: 00010046
RAX: ffff8802187acec0 RBX: ffff880087755000 RCX: 0000000000000000
RDX: ffff8802187acec8 RSI: 000000000000000a RDI: ffff8802149ad400
RBP: ffff8800c6ed9dd8 R08: 0000000000000000 R09: ffff88021b9aed10
R10: ffff88021b9aed18 R11: ffffffff81c0cbf0 R12: ffffffff81c0c9c0
R13: ffff8802149ad400 R14: ffff88021b9aed10 R15: ffffffff81c0cbe0
FS: 0000000000000000(0000) GS:ffff88022bc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000001a0b000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/u:2 (pid: 15954, threadinfo ffff8800c6ed8000, task ffff880092111730)
Stack:
ffff880087755000 ffffffff81c0c9c0 ffff8800c6ed9e38 ffffffff8109a3b9
ffff8802220f0000 00ffffff81a13410 ffffffffa026030b ffff8802149ad4a5
ffffffff81c0cbe0 ffff880087755000 ffffffff81c0cbe0 ffff880087755020
Call Trace:
[<ffffffff8109a3b9>] process_one_work+0x26c/0x27b
[<ffffffffa026030b>] ? ieee80211_netdev_select_queue+0x12/0x12 [mac80211]
[<ffffffff8109c6a8>] worker_thread+0x158/0x255
[<ffffffff8109c550>] ? manage_workers+0x26e/0x26e
[<ffffffff810a0084>] kthread+0xbf/0xc7
[<ffffffff81523dfe>] ? schedule+0x5f/0x61
[<ffffffff8109ffc5>] ? __init_kthread_worker+0x37/0x37
[<ffffffff81529b7c>] ret_from_fork+0x7c/0xb0
[<ffffffff8109ffc5>] ? __init_kthread_worker+0x37/0x37
Code: 8b 47 54 48 8b 57 60 ff c8 48 39 ca 89 47 54 74 6e 3b 47 58 7d 69 4c 8b 42 f8 31 c9 48 8d 42 f8 41 f6 c0 04 74 05 4c 89 c1 30 c9 <4c> 8b 09 4c 8b 40 08 4d
8d 61 10 49 83 e8 08 eb 32 4c 8b 58 10
RIP [<ffffffff81099d58>] cwq_dec_nr_in_flight+0x46/0xd5
RSP <ffff8800c6ed9dc8>
CR2: 0000000000000000
---[ end trace 5f3eec36f8009bcd ]---
note: kworker/u:2[15954] exited with preempt_count 1
BUG: unable to handle kernel paging request at ffffffffffffffc8
IP: [<ffffffff8109fadc>] kthread_data+0xb/0x11
PGD 1a0d067 PUD 1a0e067 PMD 0
Oops: 0000 [#2] PREEMPT SMP
Modules linked in: nfsv3 nfs_acl nfnetlink_log nfnetlink bluetooth nfsv4 auth_rpcgss nfs fscache nf_nat_ipv4 nf_nat fuse 8021q garp stp llc macvlan lockd
wanlink(O) sunrpc pktgen f71882fg coretemp hwmon mperf iTCO_wdt iTCO_vendor_support kvm cdc_acm ppdev gpio_ich snd_hda_codec_realtek i2c_i801 microcode
serio_raw lpc_ich pcspkr ath9k ath9k_common ath9k_hw ath mac80211 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e snd_page_alloc
snd_timer snd soundcore cfg80211 parport_pc parport uinput ipv6 i915 video i2c_algo_bit drm_kms_helper drm i2c_core [last unloaded: iptable_nat]
CPU 1
Pid: 15954, comm: kworker/u:2 Tainted: G D WC O 3.7.6+ #65 To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: 0010:[<ffffffff8109fadc>] [<ffffffff8109fadc>] kthread_data+0xb/0x11
RSP: 0018:ffff8800c6ed9998 EFLAGS: 00010092
RAX: 0000000000000000 RBX: ffff88022bc92b80 RCX: ffff880092111778
RDX: ffffffff81c0d460 RSI: 0000000000000001 RDI: ffff880092111730
RBP: ffff8800c6ed9998 R08: 00000000bb51f0b4 R09: 000224203fbfdcda
R10: ffff880092111730 R11: ffff8800c6ed99f8 R12: ffff880092111af8
R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff88022bc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffffffffc8 CR3: 0000000001a0b000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/u:2 (pid: 15954, threadinfo ffff8800c6ed8000, task ffff880092111730)
Stack:
ffff8800c6ed99c8 ffffffff8109c04d ffff8800c6ed99c8 ffff88022bc92b80
ffff880092111af8 ffff8800c6ed9aa8 ffff8800c6ed9a68 ffffffff8152396d
ffff8800c6ed99f8 ffff880092111730 ffff8800c6ed8010 ffff880092111730
Call Trace:
[<ffffffff8109c04d>] wq_worker_sleeping+0x15/0x73
[<ffffffff8152396d>] __schedule+0x17f/0x561
[<ffffffff81523dfe>] schedule+0x5f/0x61
[<ffffffff8108bcfd>] do_exit+0x7d4/0x7d8
[<ffffffff81525e44>] oops_end+0xba/0xc2
[<ffffffff8103218d>] no_context+0x25a/0x269
[<ffffffff81032363>] __bad_area_nosemaphore+0x1c7/0x1e7
[<ffffffff81009866>] ? __switch_to+0x1f7/0x421
[<ffffffff81032391>] bad_area_nosemaphore+0xe/0x10
[<ffffffff81527cc2>] __do_page_fault+0x313/0x385
[<ffffffff81523d0d>] ? __schedule+0x51f/0x561
[<ffffffff81523dfe>] ? schedule+0x5f/0x61
[<ffffffff81527d3d>] do_page_fault+0x9/0xb
[<ffffffff81525388>] page_fault+0x28/0x30
[<ffffffff81099d58>] ? cwq_dec_nr_in_flight+0x46/0xd5
[<ffffffff81524a55>] ? _raw_spin_lock_irq+0x25/0x2a
[<ffffffff8109a3b9>] process_one_work+0x26c/0x27b
[<ffffffffa026030b>] ? ieee80211_netdev_select_queue+0x12/0x12 [mac80211]
[<ffffffff8109c6a8>] worker_thread+0x158/0x255
[<ffffffff8109c550>] ? manage_workers+0x26e/0x26e
[<ffffffff810a0084>] kthread+0xbf/0xc7
[<ffffffff81523dfe>] ? schedule+0x5f/0x61
[<ffffffff8109ffc5>] ? __init_kthread_worker+0x37/0x37
[<ffffffff81529b7c>] ret_from_fork+0x7c/0xb0
[<ffffffff8109ffc5>] ? __init_kthread_worker+0x37/0x37
Code: 65 48 8b 04 25 c0 c6 00 00 48 8b 80 70 03 00 00 48 89 e5 48 8b 40 b8 c9 48 c1 e8 02 83 e0 01 c3 48 8b 87 70 03 00 00 55 48 89 e5 <48> 8b 40 c8 c9 c3 48 3b
3d c7 d8 b6 00 55 48 89 e5 75 09 0f bf
RIP [<ffffffff8109fadc>] kthread_data+0xb/0x11
RSP <ffff8800c6ed9998>
CR2: ffffffffffffffc8
---[ end trace 5f3eec36f8009bce ]---
Fixing recursive fault but reboot is needed!
Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0
Shutting down cpus with NMI
panic occurred, switching back to text console
Rebooting in 10 seconds..
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 02/19/2013 02:52 PM, Ben Greear wrote:
> On 02/18/2013 02:16 PM, Johannes Berg wrote:
>> On Mon, 2013-02-18 at 14:14 -0800, Ben Greear wrote:
>>> We often see crashes in work-queue processing when deleting
>>> lots of wifi station interfaces. I'm guessing that there is probably
>>> a work item that was not properly un-registered before deleting
>>> memory. I have backported some wifi fixes from upstream, so
>>> maybe they are to blame, but in case anyone has any suggestions
>>> for places to look, please let me know.
>>
>> Enable CONFIG_DEBUG_OBJECTS and CONFIG_DEBUG_OBJECTS_WORK :)
>
> That did not catch anything.
Ahh, enabled a bunch more debugging options, and got this:
sta40: deauthenticating from 00:88:aa:88:aa:88 by local choice (reason=3)
------------[ cut here ]------------
WARNING: at /home/greearb/git/linux-3.7.dev.y/lib/debugobjects.c:261 debug_print_object+0x7c/0x8d()
Hardware name: To be filled by O.E.M.
ODEBUG: free active (active state 0) object type: work_struct hint: ieee80211_sta_monitor_work+0x0/0x14 [mac80211]
Modules linked in: nf_nat_ipv4 nf_nat 8021q garp stp llc macvlan pktgen lockd sunrpc f71882fg iTCO_wdt iTCO_vendor_support coretemp gpio_ich hwmon mperf kvm
cdc_acm snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep microcode snd_seq snd_seq_device serio_raw pcspkr snd_pcm ath9k ath9k_common ath9k_hw ath
i2c_i801 ppdev mac80211 lpc_ich cfg80211 snd_page_alloc e1000e snd_timer snd soundcore parport_pc parport uinput ipv6 i915 video i2c_algo_bit drm_kms_helper drm
i2c_core [last unloaded: iptable_nat]
Pid: 14743, comm: iw Tainted: G C O 3.7.9+ #11
Call Trace:
[<ffffffff81087ef8>] warn_slowpath_common+0x80/0x98
[<ffffffff81087fa4>] warn_slowpath_fmt+0x41/0x43
[<ffffffff812a2608>] debug_print_object+0x7c/0x8d
[<ffffffffa025f5ad>] ? ieee80211_beacon_connection_loss_work+0x88/0x88 [mac80211]
[<ffffffff812a2b9a>] ? debug_check_no_obj_freed+0x65/0x1c3
[<ffffffff812a2bca>] debug_check_no_obj_freed+0x95/0x1c3
[<ffffffff8149f465>] ? netdev_release+0x39/0x3e
[<ffffffff8114cc69>] slab_free_hook+0x70/0x79
[<ffffffff8114ea3e>] kfree+0x62/0xb7
[<ffffffff8149f465>] netdev_release+0x39/0x3e
[<ffffffff8136ad67>] device_release+0x52/0x8a
[<ffffffff812937db>] kobject_release+0x121/0x158
[<ffffffff81293612>] kobject_put+0x4c/0x50
[<ffffffff8148f0d7>] netdev_run_todo+0x25c/0x27e
[<ffffffff8149b2a0>] rtnl_unlock+0x9/0xb
[<ffffffffa01d31a7>] nl80211_post_doit+0x49/0x4e [cfg80211]
[<ffffffff814b0928>] genl_rcv_msg+0x25b/0x288
[<ffffffff814b06a3>] ? genl_lock+0x12/0x14
[<ffffffff814b06cd>] ? genl_rcv+0x28/0x28
[<ffffffff814aef13>] netlink_rcv_skb+0x3e/0x8f
[<ffffffff814b06c6>] genl_rcv+0x21/0x28
[<ffffffff814aecd1>] netlink_unicast+0xe9/0x16f
[<ffffffff814af4b3>] netlink_sendmsg+0x264/0x282
[<ffffffff8147d785>] ? rcu_read_unlock+0x5b/0x5d
[<ffffffff814784ab>] __sock_sendmsg_nosec+0x58/0x61
[<ffffffff814798e6>] __sock_sendmsg+0x3d/0x48
[<ffffffff8147995a>] sock_sendmsg+0x69/0x82
[<ffffffff8112c835>] ? might_fault+0x84/0x8b
[<ffffffff814849ce>] ? copy_from_user+0x2a/0x2c
[<ffffffff81484da0>] ? verify_iovec+0x4f/0xa3
[<ffffffff8147b8e7>] __sys_sendmsg+0x1fe/0x280
[<ffffffff810a8058>] ? up_read+0x1e/0x36
[<ffffffff8116ea14>] ? fcheck_files+0xac/0xea
[<ffffffff8116efd3>] ? fget_light+0x35/0xae
[<ffffffff8147badb>] sys_sendmsg+0x3d/0x5b
[<ffffffff815595e9>] system_call_fastpath+0x16/0x1b
---[ end trace 791ff0751a368327 ]---
Will go poke around in the code to see what I can see....
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 02/18/2013 02:16 PM, Johannes Berg wrote:
> On Mon, 2013-02-18 at 14:14 -0800, Ben Greear wrote:
>> We often see crashes in work-queue processing when deleting
>> lots of wifi station interfaces. I'm guessing that there is probably
>> a work item that was not properly un-registered before deleting
>> memory. I have backported some wifi fixes from upstream, so
>> maybe they are to blame, but in case anyone has any suggestions
>> for places to look, please let me know.
>
> Enable CONFIG_DEBUG_OBJECTS and CONFIG_DEBUG_OBJECTS_WORK :)
That did not catch anything.
I have another data point that may shed some light: The test
case actually deletes the AP (on another machine) at the same
time it starts tearing down 200 stations on the first machine.
So, we are manually tearing down stations at the same time they
start timing out due to the AP disappearing, and any un-registering
that the stations might try are probably going to hit some failure cases
because the AP is of course not answering.
I could not reproduce this on a system that continually created
and tore down 400 stations against APs that remained stable and
active....
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On 02/19/2013 03:46 PM, Ben Greear wrote:
> On 02/19/2013 03:24 PM, Ben Greear wrote:
>> On 02/19/2013 02:52 PM, Ben Greear wrote:
>>> On 02/18/2013 02:16 PM, Johannes Berg wrote:
>>>> On Mon, 2013-02-18 at 14:14 -0800, Ben Greear wrote:
>>>>> We often see crashes in work-queue processing when deleting
>>>>> lots of wifi station interfaces. I'm guessing that there is probably
>>>>> a work item that was not properly un-registered before deleting
>>>>> memory. I have backported some wifi fixes from upstream, so
>>>>> maybe they are to blame, but in case anyone has any suggestions
>>>>> for places to look, please let me know.
>>>>
>>>> Enable CONFIG_DEBUG_OBJECTS and CONFIG_DEBUG_OBJECTS_WORK :)
>>>
>>> That did not catch anything.
>
> So, maybe the problem is in the sta_quiesce logic.
>
> It cancels the work items before it stops the timers, so
> I think it could re-add the work before the timers are
> stopped??
Maybe not just this, at least. In my case, I'm never seeing the
sta_quiesce code called at all..I guess it is just for cases
where we are quiescing the hardware.
So, what is supposed to clean up these work items (and timers?)
when we are deleting an interface?
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Mon, 2013-02-18 at 14:14 -0800, Ben Greear wrote:
> We often see crashes in work-queue processing when deleting
> lots of wifi station interfaces. I'm guessing that there is probably
> a work item that was not properly un-registered before deleting
> memory. I have backported some wifi fixes from upstream, so
> maybe they are to blame, but in case anyone has any suggestions
> for places to look, please let me know.
Enable CONFIG_DEBUG_OBJECTS and CONFIG_DEBUG_OBJECTS_WORK :)
johannes
On 02/19/2013 03:24 PM, Ben Greear wrote:
> On 02/19/2013 02:52 PM, Ben Greear wrote:
>> On 02/18/2013 02:16 PM, Johannes Berg wrote:
>>> On Mon, 2013-02-18 at 14:14 -0800, Ben Greear wrote:
>>>> We often see crashes in work-queue processing when deleting
>>>> lots of wifi station interfaces. I'm guessing that there is probably
>>>> a work item that was not properly un-registered before deleting
>>>> memory. I have backported some wifi fixes from upstream, so
>>>> maybe they are to blame, but in case anyone has any suggestions
>>>> for places to look, please let me know.
>>>
>>> Enable CONFIG_DEBUG_OBJECTS and CONFIG_DEBUG_OBJECTS_WORK :)
>>
>> That did not catch anything.
So, maybe the problem is in the sta_quiesce logic.
It cancels the work items before it stops the timers, so
I think it could re-add the work before the timers are
stopped??
void ieee80211_sta_quiesce(struct ieee80211_sub_if_data *sdata)
{
struct ieee80211_if_managed *ifmgd = &sdata->u.mgd;
/*
* we need to use atomic bitops for the running bits
* only because both timers might fire at the same
* time -- the code here is properly synchronised.
*/
cancel_work_sync(&ifmgd->request_smps_work);
sdata_err(sdata, "Canceling monitor_work in sta_quiesce.\n");
cancel_work_sync(&ifmgd->monitor_work);
cancel_work_sync(&ifmgd->beacon_connection_loss_work);
cancel_work_sync(&ifmgd->csa_connection_drop_work);
if (del_timer_sync(&ifmgd->timer))
set_bit(TMR_RUNNING_TIMER, &ifmgd->timers_running);
cancel_work_sync(&ifmgd->chswitch_work);
if (del_timer_sync(&ifmgd->chswitch_timer))
set_bit(TMR_RUNNING_CHANSW, &ifmgd->timers_running);
/* these will just be re-established on connection */
del_timer_sync(&ifmgd->conn_mon_timer);
del_timer_sync(&ifmgd->bcn_mon_timer);
}
>
> Ahh, enabled a bunch more debugging options, and got this:
>
> sta40: deauthenticating from 00:88:aa:88:aa:88 by local choice (reason=3)
> ------------[ cut here ]------------
> WARNING: at /home/greearb/git/linux-3.7.dev.y/lib/debugobjects.c:261 debug_print_object+0x7c/0x8d()
> Hardware name: To be filled by O.E.M.
> ODEBUG: free active (active state 0) object type: work_struct hint: ieee80211_sta_monitor_work+0x0/0x14 [mac80211]
> Modules linked in: nf_nat_ipv4 nf_nat 8021q garp stp llc macvlan pktgen lockd sunrpc f71882fg iTCO_wdt iTCO_vendor_support coretemp gpio_ich hwmon mperf kvm
> cdc_acm snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep microcode snd_seq snd_seq_device serio_raw pcspkr snd_pcm ath9k ath9k_common ath9k_hw ath
> i2c_i801 ppdev mac80211 lpc_ich cfg80211 snd_page_alloc e1000e snd_timer snd soundcore parport_pc parport uinput ipv6 i915 video i2c_algo_bit drm_kms_helper drm
> i2c_core [last unloaded: iptable_nat]
> Pid: 14743, comm: iw Tainted: G C O 3.7.9+ #11
> Call Trace:
> [<ffffffff81087ef8>] warn_slowpath_common+0x80/0x98
> [<ffffffff81087fa4>] warn_slowpath_fmt+0x41/0x43
> [<ffffffff812a2608>] debug_print_object+0x7c/0x8d
> [<ffffffffa025f5ad>] ? ieee80211_beacon_connection_loss_work+0x88/0x88 [mac80211]
> [<ffffffff812a2b9a>] ? debug_check_no_obj_freed+0x65/0x1c3
> [<ffffffff812a2bca>] debug_check_no_obj_freed+0x95/0x1c3
> [<ffffffff8149f465>] ? netdev_release+0x39/0x3e
> [<ffffffff8114cc69>] slab_free_hook+0x70/0x79
> [<ffffffff8114ea3e>] kfree+0x62/0xb7
> [<ffffffff8149f465>] netdev_release+0x39/0x3e
> [<ffffffff8136ad67>] device_release+0x52/0x8a
> [<ffffffff812937db>] kobject_release+0x121/0x158
> [<ffffffff81293612>] kobject_put+0x4c/0x50
> [<ffffffff8148f0d7>] netdev_run_todo+0x25c/0x27e
> [<ffffffff8149b2a0>] rtnl_unlock+0x9/0xb
> [<ffffffffa01d31a7>] nl80211_post_doit+0x49/0x4e [cfg80211]
> [<ffffffff814b0928>] genl_rcv_msg+0x25b/0x288
> [<ffffffff814b06a3>] ? genl_lock+0x12/0x14
> [<ffffffff814b06cd>] ? genl_rcv+0x28/0x28
> [<ffffffff814aef13>] netlink_rcv_skb+0x3e/0x8f
> [<ffffffff814b06c6>] genl_rcv+0x21/0x28
> [<ffffffff814aecd1>] netlink_unicast+0xe9/0x16f
> [<ffffffff814af4b3>] netlink_sendmsg+0x264/0x282
> [<ffffffff8147d785>] ? rcu_read_unlock+0x5b/0x5d
> [<ffffffff814784ab>] __sock_sendmsg_nosec+0x58/0x61
> [<ffffffff814798e6>] __sock_sendmsg+0x3d/0x48
> [<ffffffff8147995a>] sock_sendmsg+0x69/0x82
> [<ffffffff8112c835>] ? might_fault+0x84/0x8b
> [<ffffffff814849ce>] ? copy_from_user+0x2a/0x2c
> [<ffffffff81484da0>] ? verify_iovec+0x4f/0xa3
> [<ffffffff8147b8e7>] __sys_sendmsg+0x1fe/0x280
> [<ffffffff810a8058>] ? up_read+0x1e/0x36
> [<ffffffff8116ea14>] ? fcheck_files+0xac/0xea
> [<ffffffff8116efd3>] ? fget_light+0x35/0xae
> [<ffffffff8147badb>] sys_sendmsg+0x3d/0x5b
> [<ffffffff815595e9>] system_call_fastpath+0x16/0x1b
> ---[ end trace 791ff0751a368327 ]---
>
>
> Will go poke around in the code to see what I can see....
>
>
> Thanks,
> Ben
>
>
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com