Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752333Ab0ASFr1 (ORCPT ); Tue, 19 Jan 2010 00:47:27 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752029Ab0ASFr0 (ORCPT ); Tue, 19 Jan 2010 00:47:26 -0500 Received: from mta1.srv.hcvlny.cv.net ([167.206.4.196]:49519 "EHLO mta1.srv.hcvlny.cv.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751305Ab0ASFrZ (ORCPT ); Tue, 19 Jan 2010 00:47:25 -0500 Date: Tue, 19 Jan 2010 00:46:24 -0500 From: Michael Breuer Subject: Re: [PATCH] af_packet: Don't use skb after dev_queue_xmit() In-reply-to: <4B54E4EF.8080602@majjas.com> To: Jarek Poplawski Cc: Stephen Hemminger , David Miller , akpm@linux-foundation.org, flyboy@gmail.com, linux-kernel@vger.kernel.org, netdev@vger.kernel.org Message-id: <4B554730.6090000@majjas.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT References: <4B53906B.2020608@majjas.com> <20100117230531.GC3161@del.dom.local> <4B539A0A.2000504@majjas.com> <20100118073018.GA6270@ff.dom.local> <4B548C6B.10607@majjas.com> <20100118204658.GC3157@del.dom.local> <4B54CB0D.5070604@majjas.com> <20100118212516.GE3157@del.dom.local> <4B54D50C.90608@majjas.com> <20100118220814.GG3157@del.dom.local> <20100118221747.GH3157@del.dom.local> <4B54E4EF.8080602@majjas.com> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091204 Lightning/1.0b2pre Thunderbird/3.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4625 Lines: 88 On 1/18/2010 5:47 PM, Michael Breuer wrote: > On 1/18/2010 5:17 PM, Jarek Poplawski wrote: >> On Mon, Jan 18, 2010 at 11:08:14PM +0100, Jarek Poplawski wrote: >>> Btw, I wonder if you could test it skipping the (HP?) switch? >> If so, then of course don't forget to try tcpdump on the router. >> >> Jarek P. > Well - no.... but I'm not sure that would show anything. > > Setup diagram: > > Server->gb switch-> (100mb) wifi router -> devices > | > Win7 PC (gb) > > The problem does not occur (at least I haven't been able to recreate > it) at 100mb, and the wifi router doesn't do 1Gb. I drive the traffic > from the win7 PC to the server. I've seen the loss when the only > traffic going through the wifi router was ping & dhcp. I've also never > seen any loss on a device directly attached to the 1GB switch. I can > drive load through the wifi router while driving load from the Win7 > box, but don't see TX packet loss at all when not doing DHCP > RELEASE/RENEW. > > As there is no packet loss to devices not involved in the DHCP > sequence through the same path, I'm not really sure that the GB switch > is implicated. > > As I don't have a standalone sniffer, I'm thinking that it might be > easier to instrument places where the TX packet could be dropped and > see at least whether it's getting to the card. > > Given the circumstances of the TX drop, and that it was DHCP traffic > while under load that caused the oops rectified with the two patches, > I'm thinking that the packet loss is the current manifestation of > whatever the underlying problem is. Given the extra hop required to > break things, and given that a dhcp release/renew seems to trigger > things, I keep coming back to arp logic as being somehow implicated. > > If arp is somehow involved, then I'd expect to see manifestations > under similar circumstances with other drivers. As the pskb_may_pull > patch stopped the crash, perhaps other drivers do suffer packet loss > and it's just not been widely noticed or attributed to the kernel - > especially if the network topology is a factor. I do know people at > large enterprises who have been complaining of what *could* be this > same issue, however they're currently blaming their switch vendors. As > most traffic is TCP, this is really only noticed by those few places > deeply concerned with latency. It's likely something altogether > different, but then again, maybe not. Ok - one last update for a while ...not sure what's next... I put some printk's into sky2.c xmit logic - the packets are being sent to the card, and the i/o's are completing successfully. So it would seem either the switch is dropping the packets, or else the wifi router is. As tcpdump doesn't show the packets arriving on the wifi router, I'm leaning towards the switch. I ran wireshark on the win7 box to see what is coming off the switch. I did notice one thing that's visible to the win7 box but is not showing up on the linux wireshark - before every successful dhcpoffer, there's an XID message broadcast from the device. I'm wondering why I don't see this on the linux side: The packet is from the mac of the device, dst ff:ff:ff:ff:ff:ff; protocol eth:llc... hex packet: ffffffffffff001cccf39ff600060001af810100. Now I guess I've got some reading to do... I've got no idea what the correct application of llc messages would be given my topology :(. I do suspect that the llc stuff (or lack thereof under some conditions) is causing the switch to fail to forward the dhcpoffer message. As the dhcpoffer message is not broadcast, but directed to the remote mac address and as that address is not connected directly to the switch, I'm guessing that under some conditions whatever tells the switch how to find the mac is missing. I'd guess that the wifi router should be letting the switch know around the time it forwards the first arp and/or DHCP broadcast message from the client... or maybe the linux box should be doing something before the offer. So net-net, as far as my TX packet loss issue, sky2 is in the clear. If something on the linux side should be informing the switch about something then there may still be an issue. If the wifi router should be doing something differently, then it's unfortunately likely a 2.4.37 kernel issue (That's what dd-wrt is using). -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/