Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759261AbZJMJYd (ORCPT ); Tue, 13 Oct 2009 05:24:33 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1755851AbZJMJYc (ORCPT ); Tue, 13 Oct 2009 05:24:32 -0400 Received: from host179-6-static.238-77-b.business.telecomitalia.it ([77.238.6.179]:55104 "EHLO mail.navynet.it" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754353AbZJMJYZ (ORCPT ); Tue, 13 Oct 2009 05:24:25 -0400 X-Greylist: delayed 739 seconds by postgrey-1.27 at vger.kernel.org; Tue, 13 Oct 2009 05:24:24 EDT Message-ID: <4AD44435.3050703@navynet.it> Date: Tue, 13 Oct 2009 11:11:17 +0200 From: Massimo Cetra User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090707) MIME-Version: 1.0 To: David Miller CC: rjw@sisk.pl, linux-kernel@vger.kernel.org, kernel-testers@vger.kernel.org, netdev@vger.kernel.org Subject: Re: [Bug #14378] Problems with net/core/skbuff.c References:

<20091012.034224.64980795.davem@davemloft.net> In-Reply-To: <20091012.034224.64980795.davem@davemloft.net> Content-Type: multipart/mixed; boundary="------------070700000508040107060507" X-auth-from: mcetra@navynet.it Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 124507 Lines: 1699 This is a multi-part message in MIME format. --------------070700000508040107060507 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit David Miller ha scritto: > From: "Rafael J. Wysocki" > Date: Mon, 12 Oct 2009 00:22:04 +0200 (CEST) > > >> Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=14378 >> Subject : Problems with net/core/skbuff.c >> Submitter : Massimo Cetra >> Date : 2009-10-08 14:51 (4 days old) >> References : http://marc.info/?l=linux-kernel&m=125501488220358&w=4 >> > > I don't know what to do about this one. > > The user indicates that they have the vserver patches applied, > so maybe there is some interaction with that stuff. > Actually i found another oops which is very similar to the previous one. Here, vserver is not involved, and the problem starts at drbd which lives in kernel space (the other oops started at ocfs2). Both ocfs2 and drbd make heavy use of network I/O so i guess the problem is something in the network layer. Anything i can do to help to debugging and solving this issue ? Thanks Max --------------070700000508040107060507 Content-Type: text/plain; name="oops2.txt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="oops2.txt" [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? __kfree_skb+0x9/0x7d [60257.728500] [] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [] ? __call_console_drivers+0x5e/0x6f [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? release_console_sem+0x115/0x1ba [60257.728500] [] ? vprintk+0x2f2/0x34b [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? printk+0x4e/0x5d [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? getnstimeofday+0x55/0xaf [60257.728500] [] ? ktime_get_ts+0x21/0x49 [60257.728500] [] ? ktime_get+0xc/0x41 [60257.728500] [] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [] ? apic_timer_interrupt+0x13/0x20 [60257.728500] [] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [] ? finish_task_switch+0x5b/0xec [60257.728500] [] ? thread_return+0x47/0xd5 [60257.728500] [] ? tcp_current_mss+0x3f/0x5a [60257.728500] [] ? schedule_timeout+0x21/0x197 [60257.728500] [] ? local_bh_disable+0xe/0x10 [60257.728500] [] ? _spin_lock_bh+0x13/0x29 [60257.728500] [] ? release_sock+0x19/0xc3 [60257.728500] [] ? sk_wait_data+0x85/0xca [60257.728500] [] ? autoremove_wake_function+0x0/0x2e [60257.728500] [] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [] ? tcp_recvmsg+0x421/0xabd [60257.728500] [] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [] ? sock_recvmsg+0xf7/0x18d [60257.728500] [] ? autoremove_wake_function+0x0/0x2e [60257.728500] [] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [] ? child_rip+0xa/0x20 [60257.728500] [] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f2c ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? __kfree_skb+0x9/0x7d [60257.728500] [] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [] ? __call_console_drivers+0x5e/0x6f [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? release_console_sem+0x115/0x1ba [60257.728500] [] ? vprintk+0x2f2/0x34b [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? printk+0x4e/0x5d [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? getnstimeofday+0x55/0xaf [60257.728500] [] ? ktime_get_ts+0x21/0x49 [60257.728500] [] ? ktime_get+0xc/0x41 [60257.728500] [] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [] ? apic_timer_interrupt+0x13/0x20 [60257.728500] [] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [] ? finish_task_switch+0x5b/0xec [60257.728500] [] ? thread_return+0x47/0xd5 [60257.728500] [] ? tcp_current_mss+0x3f/0x5a [60257.728500] [] ? schedule_timeout+0x21/0x197 [60257.728500] [] ? local_bh_disable+0xe/0x10 [60257.728500] [] ? _spin_lock_bh+0x13/0x29 [60257.728500] [] ? release_sock+0x19/0xc3 [60257.728500] [] ? sk_wait_data+0x85/0xca [60257.728500] [] ? autoremove_wake_function+0x0/0x2e [60257.728500] [] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [] ? tcp_recvmsg+0x421/0xabd [60257.728500] [] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [] ? sock_recvmsg+0xf7/0x18d [60257.728500] [] ? autoremove_wake_function+0x0/0x2e [60257.728500] [] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [] ? child_rip+0xa/0x20 [60257.728500] [] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f2d ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? __kfree_skb+0x9/0x7d [60257.728500] [] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [] ? __call_console_drivers+0x5e/0x6f [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? release_console_sem+0x115/0x1ba [60257.728500] [] ? vprintk+0x2f2/0x34b [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? printk+0x4e/0x5d [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? getnstimeofday+0x55/0xaf [60257.728500] [] ? ktime_get_ts+0x21/0x49 [60257.728500] [] ? ktime_get+0xc/0x41 [60257.728500] [] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [] ? apic_timer_interrupt+0x13/0x20 [60257.728500] [] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [] ? finish_task_switch+0x5b/0xec [60257.728500] [] ? thread_return+0x47/0xd5 [60257.728500] [] ? tcp_current_mss+0x3f/0x5a [60257.728500] [] ? schedule_timeout+0x21/0x197 [60257.728500] [] ? local_bh_disable+0xe/0x10 [60257.728500] [] ? _spin_lock_bh+0x13/0x29 [60257.728500] [] ? release_sock+0x19/0xc3 [60257.728500] [] ? sk_wait_data+0x85/0xca [60257.728500] [] ? autoremove_wake_function+0x0/0x2e [60257.728500] [] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [] ? tcp_recvmsg+0x421/0xabd [60257.728500] [] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [] ? sock_recvmsg+0xf7/0x18d [60257.728500] [] ? autoremove_wake_function+0x0/0x2e [60257.728500] [] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [] ? child_rip+0xa/0x20 [60257.728500] [] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f2e ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? __kfree_skb+0x9/0x7d [60257.728500] [] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [] ? __call_console_drivers+0x5e/0x6f [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? release_console_sem+0x115/0x1ba [60257.728500] [] ? vprintk+0x2f2/0x34b [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? printk+0x4e/0x5d [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? getnstimeofday+0x55/0xaf [60257.728500] [] ? ktime_get_ts+0x21/0x49 [60257.728500] [] ? ktime_get+0xc/0x41 [60257.728500] [] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [] ? apic_timer_interrupt+0x13/0x20 [60257.728500] [] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [] ? finish_task_switch+0x5b/0xec [60257.728500] [] ? thread_return+0x47/0xd5 [60257.728500] [] ? tcp_current_mss+0x3f/0x5a [60257.728500] [] ? schedule_timeout+0x21/0x197 [60257.728500] [] ? local_bh_disable+0xe/0x10 [60257.728500] [] ? _spin_lock_bh+0x13/0x29 [60257.728500] [] ? release_sock+0x19/0xc3 [60257.728500] [] ? sk_wait_data+0x85/0xca [60257.728500] [] ? autoremove_wake_function+0x0/0x2e [60257.728500] [] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [] ? tcp_recvmsg+0x421/0xabd [60257.728500] [] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [] ? sock_recvmsg+0xf7/0x18d [60257.728500] [] ? autoremove_wake_function+0x0/0x2e [60257.728500] [] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [] ? child_rip+0xa/0x20 [60257.728500] [] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f2f ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? __kfree_skb+0x9/0x7d [60257.728500] [] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [] ? __call_console_drivers+0x5e/0x6f [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? release_console_sem+0x115/0x1ba [60257.728500] [] ? vprintk+0x2f2/0x34b [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? printk+0x4e/0x5d [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? getnstimeofday+0x55/0xaf [60257.728500] [] ? ktime_get_ts+0x21/0x49 [60257.728500] [] ? ktime_get+0xc/0x41 [60257.728500] [] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [] ? apic_timer_interrupt+0x13/0x20 [60257.728500] [] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [] ? finish_task_switch+0x5b/0xec [60257.728500] [] ? thread_return+0x47/0xd5 [60257.728500] [] ? tcp_current_mss+0x3f/0x5a [60257.728500] [] ? schedule_timeout+0x21/0x197 [60257.728500] [] ? local_bh_disable+0xe/0x10 [60257.728500] [] ? _spin_lock_bh+0x13/0x29 [60257.728500] [] ? release_sock+0x19/0xc3 [60257.728500] [] ? sk_wait_data+0x85/0xca [60257.728500] [] ? autoremove_wake_function+0x0/0x2e [60257.728500] [] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [] ? tcp_recvmsg+0x421/0xabd [60257.728500] [] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [] ? sock_recvmsg+0xf7/0x18d [60257.728500] [] ? autoremove_wake_function+0x0/0x2e [60257.728500] [] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [] ? child_rip+0xa/0x20 [60257.728500] [] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f30 ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? __kfree_skb+0x9/0x7d [60257.728500] [] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [] ? __call_console_drivers+0x5e/0x6f [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? release_console_sem+0x115/0x1ba [60257.728500] [] ? vprintk+0x2f2/0x34b [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? printk+0x4e/0x5d [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? getnstimeofday+0x55/0xaf [60257.728500] [] ? ktime_get_ts+0x21/0x49 [60257.728500] [] ? ktime_get+0xc/0x41 [60257.728500] [] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [] ? apic_timer_interrupt+0x13/0x20 [60257.728500] [] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [] ? finish_task_switch+0x5b/0xec [60257.728500] [] ? thread_return+0x47/0xd5 [60257.728500] [] ? tcp_current_mss+0x3f/0x5a [60257.728500] [] ? schedule_timeout+0x21/0x197 [60257.728500] [] ? local_bh_disable+0xe/0x10 [60257.728500] [] ? _spin_lock_bh+0x13/0x29 [60257.728500] [] ? release_sock+0x19/0xc3 [60257.728500] [] ? sk_wait_data+0x85/0xca [60257.728500] [] ? autoremove_wake_function+0x0/0x2e [60257.728500] [] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [] ? tcp_recvmsg+0x421/0xabd [60257.728500] [] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [] ? sock_recvmsg+0xf7/0x18d [60257.728500] [] ? autoremove_wake_function+0x0/0x2e [60257.728500] [] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [] ? child_rip+0xa/0x20 [60257.728500] [] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f31 ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? __kfree_skb+0x9/0x7d [60257.728500] [] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [] ? __call_console_drivers+0x5e/0x6f [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? release_console_sem+0x115/0x1ba [60257.728500] [] ? vprintk+0x2f2/0x34b [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? printk+0x4e/0x5d [60257.728500] [] ? kvm_clock_read+0x4d/0x52 [60257.728500] [] ? getnstimeofday+0x55/0xaf [60257.728500] [] ? ktime_get_ts+0x21/0x49 [60257.728500] [] ? ktime_get+0xc/0x41 [60257.728500] [] ? hrtimer_interrupt+0x9c/0x146 [60257.728500] [] ? smp_apic_timer_interrupt+0x80/0x93 [60257.728500] [] ? apic_timer_interrupt+0x13/0x20 [60257.728500] [] ? cap_socket_recvmsg+0x0/0x3 [60257.728500] [] ? _spin_unlock_irq+0xd/0x31 [60257.728500] [] ? finish_task_switch+0x5b/0xec [60257.728500] [] ? thread_return+0x47/0xd5 [60257.728500] [] ? tcp_current_mss+0x3f/0x5a [60257.728500] [] ? schedule_timeout+0x21/0x197 [60257.728500] [] ? local_bh_disable+0xe/0x10 [60257.728500] [] ? _spin_lock_bh+0x13/0x29 [60257.728500] [] ? release_sock+0x19/0xc3 [60257.728500] [] ? sk_wait_data+0x85/0xca [60257.728500] [] ? autoremove_wake_function+0x0/0x2e [60257.728500] [] ? tcp_prequeue_process+0x9d/0xac [60257.728500] [] ? tcp_recvmsg+0x421/0xabd [60257.728500] [] ? cfq_add_rq_rb+0xd2/0xe5 [60257.728500] [] ? sock_common_recvmsg+0x30/0x45 [60257.728500] [] ? sock_recvmsg+0xf7/0x18d [60257.728500] [] ? autoremove_wake_function+0x0/0x2e [60257.728500] [] ? _spin_lock_irqsave+0x25/0x41 [60257.728500] [] ? _drbd_send_cmd+0x11d/0x1eb [drbd] [60257.728500] [] ? blk_rq_map_sg+0x12d/0x276 [60257.728500] [] ? drbd_recv+0x74/0x147 [drbd] [60257.728500] [] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd] [60257.728500] [] ? drbd_recv_header+0x18/0xc8 [drbd] [60257.728500] [] ? drbdd+0x2c/0x1d6 [drbd] [60257.728500] [] ? drbdd_init+0xf2/0x14a [drbd] [60257.728500] [] ? drbd_thread_setup+0x16f/0x231 [drbd] [60257.728500] [] ? child_rip+0xa/0x20 [60257.728500] [] ? vgacon_cursor+0x0/0x1a4 [60257.728500] [] ? drbd_thread_setup+0x0/0x231 [drbd] [60257.728500] [] ? child_rip+0x0/0x20 [60257.728500] ---[ end trace 694817acca794f32 ]--- [60257.728500] ------------[ cut here ]------------ [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8() [60257.728500] Hardware name: [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1 [60257.728500] Call Trace: [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? warn_slowpath_common+0x77/0xa3 [60257.728500] [] ? skb_release_head_state+0x64/0xc8 [60257.728500] [] ? __kfree_skb+0x9/0x7d [60257.728500] [] ? free_old_xmit_skbs+0x51/0x6e [virtio_net] [60257.728500] [] ? start_xmit+0x26/0xf2 [virtio_net] [60257.728500] [] ? netpoll_send_skb+0xd2/0x205 [60257.728500] [] ? write_msg+0x90/0xeb [netconsole] [60257.728500] [] ? __call_console_drivers+0x5e/0x6f [60257.728500] [