Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755674AbXHBJXu (ORCPT ); Thu, 2 Aug 2007 05:23:50 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1754037AbXHBJX2 (ORCPT ); Thu, 2 Aug 2007 05:23:28 -0400 Received: from courier.cs.helsinki.fi ([128.214.9.1]:43182 "EHLO mail.cs.helsinki.fi" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753455AbXHBJXZ (ORCPT ); Thu, 2 Aug 2007 05:23:25 -0400 Date: Thu, 2 Aug 2007 12:23:23 +0300 (EEST) From: "=?ISO-8859-1?Q?Ilpo_J=E4rvinen?=" X-X-Sender: ijjarvin@kivilampi-30.cs.helsinki.fi To: "Darryl L. Miles" cc: LKML , Netdev Subject: Re: TCP SACK issue, hung connection, tcpdump included In-Reply-To: <46AEC286.2030302@netbauds.net> Message-ID: References: <46AC2CBE.5010500@netbauds.net> <20070729064511.GA18718@1wt.eu> <20070729085427.GA22784@1wt.eu> <20070729160721.GA31276@1wt.eu> <46AEC286.2030302@netbauds.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4936 Lines: 107 On Tue, 31 Jul 2007, Darryl L. Miles wrote: > I've been able to capture a tcpdump from both ends during the problem and its > my belief there is a bug in 2.6.20.1 (at the client side) in that it issues a > SACK option for an old sequence which the current window being advertised is > beyond it. This is the most concerning issue as the integrity of the sequence > numbers doesn't seem right (to my limited understanding anyhow). You probably didn't check the reference I explicitly gave to those who are not familiar how DSACK works, just in case you didn't pick it up last time, here it is again for you: RFC2883... However, if DSACKs really bother you still (though it shouldn't :-)), IIRC I also told you how you're able to turn it off (tcp_dsack sysctl) but I assure you that it's not a bug but feature called DSACK [RFC2883], there's _absolutely_ nothing wrong with it, instead, it would be wrong to _not_ send the below snd_una SACK in this scenario when tcp_dsack set to 1. > There is another concern of why the SERVER performed a retransmission in the > first place, when the tcpdump shows the ack covering it has been seen. There are only three possible reasons to this thing: 1) The ACK didn't reach the SERVER (your logs prove this to not be the case) 2) The ACK got discarded by the SERVER 3) The SERVER (not the client) is buggy and sends an incorrect retransmission ...So we have just two options remaining... > I have made available the full dumps at: > > http://darrylmiles.org/snippets/lkml/20070731/ Thanks about these... Based on a quick check, it is rather clear that the SERVER is for some reason discarding the packets it's receiving: 04:11:26.833935 IP CLIENT.43726 > SERVER.ssh: P 4239:4287(48) ack 28176 win 501 04:11:27.132425 IP SERVER.ssh > CLIENT.43726: . 26016:27464(1448) ack 4239 win 2728 04:11:27.230081 IP CLIENT.43726 > SERVER.ssh: . ack 28176 win 501 This sequence is interesting from the client side: > > 03:58:56.419034 IP SERVER.ssh > CLIENT.43726: . 26016:27464(1448) ack 4239 > win 2728 # S1 > 03:58:56.419100 IP CLIENT.43726 > SERVER.ssh: . ack 27464 win 501 > # C1 > 03:58:56.422019 IP SERVER.ssh > CLIENT.43726: P 27464:28176(712) ack 4239 > win 2728 # S2 > 03:58:56.422078 IP CLIENT.43726 > SERVER.ssh: . ack 28176 win 501 > # C2 > > The above 4 packets look as expect to me. Then we suddenly see a > retransmission of 26016:27464. > > 03:58:56.731597 IP SERVER.ssh > CLIENT.43726: . 26016:27464(1448) ack 4239 > win 2728 # S3 ...Look at this on the retransmission: ... timestamp 16346128 819458864> ...it tells us what really got received by the TCP. The corresponding ACK with matching timestamp is, surprise, surprise, this one: > 03:58:56.340180 IP CLIENT.43726 > SERVER.ssh: . ack 26016 win 501 > ...thus the SERVER has _not_ received but discarded the subsequent cumulative ACKs!!! Therefore it's retransmitting from 26016 onward but never receives any reply as everything seems to get discarded... There was one bad checksum btw: > 03:58:56.365662 IP (tos 0x10, ttl 64, id 28685, offset 0, flags [DF], > proto 6, length: 764) SERVER.ssh > CLIENT.43726: P [bad tcp cksum 6662 > (->ef2b)!] 617734888:617735600(712) ack 2634113543 win 2728 > > There are some changes in 2.6.22 that appear to affect TCP SACK handling > does this fix a known issue ? There is no such "known issue" :-)... This issue has nothing to do with TCP SACK handling, since that code _won't_ be reached... We could verify that from the timestamps. But if you still insist that SACK under snd_una is the issue, please turn tcp_dsack to 0 on the CLIENT, you will not get them after that and you can be happy as your out-of-window SACK "issue" is then fixed :-)... ...Seriously, somebody else than me is probably better in suggesting what could cause the discarding at the SERVER in this case. SNMP stuff Dave was asking could help, you can find them from /proc/net/{netstat,snmp}... -- i. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/