Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757196AbZDGDDc (ORCPT ); Mon, 6 Apr 2009 23:03:32 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752247AbZDGDDU (ORCPT ); Mon, 6 Apr 2009 23:03:20 -0400 Received: from Mycroft.westnet.com ([216.187.52.7]:51326 "EHLO Mycroft.westnet.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752230AbZDGDDT (ORCPT ); Mon, 6 Apr 2009 23:03:19 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18906.49779.243047.207639@stoffel.org> Date: Mon, 6 Apr 2009 23:03:15 -0400 From: "John Stoffel" To: linux-kernel@vger.kernel.org CC: netdev@vger.kernel.org Subject: tcpdump crashes 2.6.29.1 (and 2.6.29-rc5) X-Mailer: VM 8.0.9 under Emacs 22.3.1 (i486-pc-linux-gnu) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 13048 Lines: 225 Hi, I've got an AMD64x2 system, x86_64, SMP, PREEMPT, with 4Gb of RAM, running 2.6.29.1 (most recently running 2.6.29-rc5 for a while before that) which crashes when I run 'tcpdump host x.y.z.n'. I'm on linux-kernel, but not netdev, so please keep me in the headers. I have't tried wireshark or ethereal yet, since I just wanted to see if packets were getting forwarded. My main firewall is a m0n0wall WRAP board, so there's no much to do there debugging wise. Ping does seem to work from both the firewall (192.1681.254) to 192.168.2.200 once I put in the proper static route via sail (192.168.1.3/192.168.2.3). Background: I'm working on setting up a PXE boot environment for a new box without a CDROM. My current system, sail, has eth0 on the 192.168.1.0/24 subnet, with an IP of 192.168.1.3, and my WAN router (with DHCP) is setup at 192.168.1.254. So I setup eth1 on the 192.168.2.0/24 subnet, with a DHCP server on my host, and an IP of 192.168.2.3 on that subnet. More details: eth0 Link encap:Ethernet HWaddr 00:1e:8c:45:f3:e5 inet addr:192.168.1.3 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8663 errors:0 dropped:0 overruns:0 frame:0 TX packets:10088 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3329485 (3.3 MB) TX bytes:2088101 (2.0 MB) Interrupt:26 Base address:0xe000 eth1 Link encap:Ethernet HWaddr 00:1e:8c:46:0e:74 inet addr:192.168.2.3 Bcast:192.168.2.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8 errors:0 dropped:0 overruns:0 frame:0 TX packets:71 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1608 (1.6 KB) TX bytes:11169 (11.1 KB) Interrupt:27 Base address:0xa000 > cat /proc/sys/net/ipv4/ip_forward 1 > cat /proc/version Linux version 2.6.29.1 (root@sail) (gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4) ) #22 SMP PREEMPT Mon Apr 6 21:41:56 EDT 2009 > netstat -nr Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1 0.0.0.0 192.168.1.254 0.0.0.0 UG 0 0 0 eth0 > ip route 192.168.2.0/24 dev eth1 proto kernel scope link src 192.168.2.3 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.3 169.254.0.0/16 dev eth1 scope link metric 1000 default via 192.168.1.254 dev eth0 metric 100 > tcpdump --version tcpdump version 3.9.8 libpcap version 1.0.0 I don't have any ip_tables stuff loaded, which might be needed for my setup, but which doesn't seem to be implicated in this crash right now. I can now PXEboot my client just fine, but connections from the 192.168.2.200 PXE host don't seem to get properly forwarded out to the internet so I can continue the network boot and install of Debian. So to figure out what I was going wrong, I would try to do 'tcpdump host 192.168.2.200' on 'sail' and it would instantly hang. I mucked around and got netconsole setup and got the following ksymoops. Hopefully they'll be helpful. I'll be happy to try and bisect this, and I'll start moving back in versions to see if I can find where this crash takes place. [ 1396.271782] BUG: unable to handle kernel paging request at ffff880020a15cc0 [ 1396.271809] IP: [] memcpy_c+0xb/0x20 [ 1396.271837] PGD 202063 PUD 206063 PMD 20a001e2 [ 1396.271864] Oops: 0000 [#1] PREEMPT SMP [ 1396.271887] last sysfs file: /sys/module/loop/initstate [ 1396.271896] CPU 1 [ 1396.271907] Modules linked in: cryptoloop loop kvm_amd netconsole configfs binfmt_misc radeon drm af_packet kvm nfs ftdi_sio usbserial ata_generic pata_jmicron ohci1394 ieee1394 k8temp uvcvideo videodev v4l1_compat v4l2_compat_ioctl32 i2c_nforce2 i2c_core [last unloaded: kvm_amd] [ 1396.272062] Pid: 0, comm: swapper Not tainted 2.6.29.1 #22 System Product Name [ 1396.272070] RIP: 0010:[] [] memcpy_c+0xb/0x20 [ 1396.272087] RSP: 0018:ffff88011e8b7b00 EFLAGS: 00010002 [ 1396.272095] RAX: ffff8800cac5d2c0 RBX: 0000000000000012 RCX: 0000000000000002 [ 1396.272103] RDX: 0000000000000002 RSI: ffff880020a15cc0 RDI: ffff8800cac5d2c0 [ 1396.272112] RBP: ffff88011e8b7b38 R08: 0000000000000080 R09: 000000000004b000 [ 1396.272120] R10: 0000000000000012 R11: 00000000ffffff82 R12: 00000000000002d2 [ 1396.272129] R13: 0000000000000012 R14: ffff880020a15cc0 R15: 0000000000000012 [ 1396.272138] FS: 00007fe36d6f1700(0000) GS:ffff88011f5d92c0(0000) knlGS:0000000000000000 [ 1396.272146] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [ 1396.272155] CR2: ffff880020a15cc0 CR3: 0000000115e14000 CR4: 00000000000006e0 [ 1396.272163] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1396.272172] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1396.272181] Process swapper (pid: 0, threadinfo ffff88011e8b0000, task ffff88011e89b820) [ 1396.272189] Stack: [ 1396.272195] ffffffff804e6444 ffff88010aab2ec0 0000000000000000 0000000000000012 [ 1396.272216] 0000000020a15cc0 00000000000002c0 ffff88010aab2ec0 ffff88011e8b7b78 [ 1396.272247] ffffffff804e760e 0000000000000096 ffff8800cac5d280 ffff88010aab2ec0 [ 1396.272279] Call Trace: [ 1396.272287] <0> [] ? mon_copy_to_buff+0x74/0xa0 [ 1396.272309] [] mon_dmapeek_vec+0x9e/0x120 [ 1396.272322] [] mon_bin_event+0x34f/0x3d0 [ 1396.272338] [] mon_bin_complete+0x10/0x20 [ 1396.272350] [] mon_bus_complete+0x48/0x80 [ 1396.272364] [] mon_complete+0x24/0x40 [ 1396.272376] [] usb_hcd_giveback_urb+0x9b/0xe0 [ 1396.272390] [] ehci_urb_done+0x106/0x130 [ 1396.272404] [] qh_completions+0x31a/0x680 [ 1396.272417] [] ? _spin_unlock_irqrestore+0x4a/0x60 [ 1396.272433] [] ehci_work+0xe3/0xbf0 [ 1396.272446] [] ehci_irq+0x2b0/0x430 [ 1396.272460] [] ? getnstimeofday+0x59/0xe0 [ 1396.272474] [] ? ktime_get_ts+0x59/0x60 [ 1396.272489] [] usb_hcd_irq+0x4c/0xb0 [ 1396.272502] [] handle_IRQ_event+0x43/0x80 [ 1396.272516] [] handle_fasteoi_irq+0x82/0x100 [ 1396.272532] [] do_IRQ+0x81/0x110 [ 1396.272545] [] ret_from_intr+0x0/0x29 [ 1396.272558] <0> [] ? native_safe_halt+0x6/0x10 [ 1396.272580] [] ? default_idle+0x4d/0x50 [ 1396.272594] [] ? c1e_idle+0x51/0x120 [ 1396.272607] [] ? atomic_notifier_call_chain+0x11/0x20 [ 1396.272620] [] ? cpu_idle+0x69/0xd0 [ 1396.272636] [] ? start_secondary+0x182/0x1db [ 1396.272650] Code: 81 ea d8 1f 00 00 48 3b 42 20 73 07 48 8b 50 f9 31 c0 c3 31 d2 48 c7 c0 f2 ff ff ff c3 90 90 90 48 89 f8 89 d1 c1 e9 03 83 e2 07 48 a5 89 d1 f3 a4 c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 [ 1396.272759] RIP [] memcpy_c+0xb/0x20 [ 1396.272759] RSP [ 1396.272759] CR2: ffff880020a15cc0 [ 1396.272759] ---[ end trace 091310ffa51c7f18 ]--- [ 1396.272759] Kernel panic - not syncing: Fatal exception in interrupt [ 1396.272759] ------------[ cut here ]------------ [ 1396.272759] WARNING: at kernel/smp.c:329 smp_call_function_many+0x231/0x2a0() [ 1396.272759] Hardware name: System Product Name [ 1396.272759] Modules linked in: cryptoloop loop kvm_amd netconsole configfs binfmt_misc radeon drm af_packet kvm nfs ftdi_sio usbserial ata_generic pata_jmicron ohci1394 ieee1394 k8temp uvcvideo videodev v4l1_compat v4l2_compat_ioctl32 i2c_nforce2 i2c_core [last unloaded: kvm_amd] [ 1396.272759] Pid: 0, comm: swapper Tainted: G D 2.6.29.1 #22 [ 1396.272759] Call Trace: [ 1396.272759] [] warn_slowpath+0xd0/0x130 [ 1396.272759] [] ? vsnprintf+0x351/0xbb0 [ 1396.272759] [] ? put_dec+0x11f/0x130 [ 1396.272759] [] ? _spin_unlock_irqrestore+0x4a/0x60 [ 1396.272759] [] ? nv_start_xmit_optimized+0x47b/0x5c0 [ 1396.272759] [] ? stop_this_cpu+0x0/0x30 [ 1396.272759] [] ? netpoll_send_udp+0x205/0x220 [ 1396.272759] [] smp_call_function_many+0x231/0x2a0 [ 1396.272759] [] ? stop_this_cpu+0x0/0x30 [ 1396.272759] [] ? default_spin_lock_flags+0x9/0x10 [ 1396.272759] [] ? _spin_unlock_irqrestore+0x4a/0x60 [ 1396.272759] [] ? _spin_unlock_irqrestore+0x4a/0x60 [ 1396.272759] [] ? release_console_sem+0x1bd/0x210 [ 1396.272759] [] ? vprintk+0x3d0/0x450 [ 1396.272759] [] ? _spin_unlock_irqrestore+0x4a/0x60 [ 1396.272759] [] ? printk+0x67/0x6a [ 1396.272759] [] ? crash_kexec+0x74/0x100 [ 1396.272759] [] ? default_spin_lock_flags+0x9/0x10 [ 1396.272759] [] ? _spin_unlock_irqrestore+0x4a/0x60 [ 1396.272759] [] ? _spin_unlock_irqrestore+0x4a/0x60 [ 1396.272759] [] ? release_console_sem+0x1bd/0x210 [ 1396.272759] [] smp_call_function+0x2f/0x70 [ 1396.272759] [] native_smp_send_stop+0x23/0x50 [ 1396.272759] [] panic+0xbe/0x183 [ 1396.272759] [] ? crash_kexec+0x74/0x100 [ 1396.272759] [] ? _spin_unlock_irqrestore+0x4a/0x60 [ 1396.272759] [] ? release_console_sem+0x1bd/0x210 [ 1396.272759] [] ? console_unblank+0x75/0x90 [ 1396.272759] [] oops_end+0xe4/0xf0 [ 1396.272759] [] do_page_fault+0x22a/0xcd0 [ 1396.272759] [] ? complete_signal+0x121/0x210 [ 1396.272759] [] ? send_signal+0x1ce/0x2d0 [ 1396.272759] [] ? check_preempt_wakeup+0x175/0x190 [ 1396.272759] [] ? try_to_wake_up+0x16c/0x240 [ 1396.272759] [] ? nv_swncq_host_interrupt+0x17b/0x760 [ 1396.272759] [] ? enqueue_task_fair+0x188/0x1d0 [ 1396.272759] [] page_fault+0x25/0x30 [ 1396.272759] [] ? memcpy_c+0xb/0x20 [ 1396.272759] [] ? mon_copy_to_buff+0x74/0xa0 [ 1396.272759] [] mon_dmapeek_vec+0x9e/0x120 [ 1396.272759] [] mon_bin_event+0x34f/0x3d0 [ 1396.272759] [] mon_bin_complete+0x10/0x20 [ 1396.272759] [] mon_bus_complete+0x48/0x80 [ 1396.272759] [] mon_complete+0x24/0x40 [ 1396.272759] [] usb_hcd_giveback_urb+0x9b/0xe0 [ 1396.272759] [] ehci_urb_done+0x106/0x130 [ 1396.272759] [] qh_completions+0x31a/0x680 [ 1396.272759] [] ? _spin_unlock_irqrestore+0x4a/0x60 [ 1396.272759] [] ehci_work+0xe3/0xbf0 [ 1396.272759] [] ehci_irq+0x2b0/0x430 [ 1396.272759] [] ? getnstimeofday+0x59/0xe0 [ 1396.272759] [] ? ktime_get_ts+0x59/0x60 [ 1396.272759] [] usb_hcd_irq+0x4c/0xb0 [ 1396.272759] [] handle_IRQ_event+0x43/0x80 [ 1396.272759] [] handle_fasteoi_irq+0x82/0x100 [ 1396.272759] [] do_IRQ+0x81/0x110 [ 1396.272759] [] ret_from_intr+0x0/0x29 [ 1396.272759] [] ? native_safe_halt+0x6/0x10 [ 1396.272759] [] ? default_idle+0x4d/0x50 [ 1396.272759] [] ? c1e_idle+0x51/0x120 [ 1396.272759] [] ? atomic_notifier_call_chain+0x11/0x20 [ 1396.272759] [] ? cpu_idle+0x69/0xd0 [ 1396.272759] [] ? start_secondary+0x182/0x1db [ 1396.272759] ---[ end trace 091310ffa51c7f19 ]--- [ 1396.272759] ------------[ cut here ]------------ [ 1396.272759] WARNING: at kernel/smp.c:226 smp_call_function_single+0xdf/0x170() [ 1396.272759] Hardware name: System Product Name [ 1396.272759] Modules linked in: cryptoloop loop kvm_amd netconsole configfs binfmt_misc radeon drm af_packet kvm nfs ftdi_sio usbserial ata_generic pata_jmicron ohci1394 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/