LinuxLists.cc - ipv6 stack seems to forget to send ACKs

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

> tornado.wiggy.net is my client running Linux 2.4.19 (unpatched, UP machine),
> and 2001:968:1::2 is the icecast server running Linux 2.4.20-rc2-ac3 (SMP).
> If you want to test the stream yourself, please stream from
> http://ipv6.lkml.org:8000/difm .
Which ipv6 client should i be using ?

Regards,
Maciej

2003-01-08 13:23:55

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Previously Maciej Soltysiak wrote:
> Which ipv6 client should i be using ?

I am using a patched version of xmms using the patch from
http://bugs.debian.org/155955 . (Don't forget to rerun autoconf
after applying the patch). If you want I can create an ipv6
enabled xmms.deb for you if you are using Debian.

Wichert.

--
Wichert Akkerman <[email protected]> http://www.wiggy.net/
A random hacker

2003-01-08 13:47:30

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

> No problem. Are you running stable, testing or unstable?
stable here.

Maciej

2003-01-08 13:42:25

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

> I am using a patched version of xmms using the patch from
> http://bugs.debian.org/155955 . (Don't forget to rerun autoconf
> after applying the patch). If you want I can create an ipv6
> enabled xmms.deb for you if you are using Debian.
Well, i have xmms from deb package, so i would rather use an
ip6 enabled deb. If that's not a problem.

Maciej

2003-01-08 13:46:18

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Previously Maciej Soltysiak wrote:
> Well, i have xmms from deb package, so i would rather use an
> ip6 enabled deb. If that's not a problem.

No problem. Are you running stable, testing or unstable?

Wichert.

--
Wichert Akkerman <[email protected]> http://www.wiggy.net/
A random hacker

2003-01-08 14:02:58

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Previously Maciej Soltysiak wrote:
> stable here.

Packages are at http://www.wiggy.net/tmp/xmms/ now.

Wichert.

--
Wichert Akkerman <[email protected]> http://www.wiggy.net/
A random hacker

2003-01-08 14:34:46

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

I had no problems listening to the stream, except a gap after about 3mins.
tcpdump showed the client closed the connection, and quickly initiated a
new one. Since then i had 15mins of nonstop playback and it stopped,
similarily to your dump.

The tcpdump is similar to yours, except i do not have traffic class info.
And rarely sack was used.

Is there a ip6 mangling router in your route to the icecast server?

I have been listening on an ip6 enabled host behind my ip6 tunnelling
router to my MAN.

Client: linux-2.4.21-pre1
Router: linux-2.4.20-grsec

I have to go now, i will look into that later.

Regards,
Maciej

2003-01-08 14:45:28

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Previously Maciej Soltysiak wrote:
> Is there a ip6 mangling router in your route to the icecast server?

Not to my knowledge. traceroute6 shows:

traceroute to ipv6.lkml.org (2001:968:1::2) from
3ffe:8280:10:1d0:290:27ff:fe2d:968c, 30 hops max, 16 byte packets
1 thunder.wiggy.net (3ffe:8280:10:1d0:250:4ff:fe0b:dd79) 0.666 ms 0.22 ms 0.199 ms
2 xs4all-29.ipv6.xs4all.nl (3ffe:8280:0:2001::58) 27.568 ms 28.012 ms 30.177 ms
3 26.ge-0-2-0.xr1.pbw.xs4all.net (2001:888:0:3::1) 22.035 ms 19.528 ms 44.644 ms
4 0.ge-0-3-0.xr1.sara.xs4all.net (2001:888:2:1::1) 19.519 ms 19.002 ms 21.974 ms
5 fe-0-0-0.ams-core-01.network6.isp-services.nl (2001:7f8:1::a502:4875:1) 19.978 ms 30.278 ms 20.248 ms
6 2001:968::2 (2001:968::2) 24.246 ms 24.083 ms 22.918 ms
7 2001:968:1::2 (2001:968:1::2) 24.978 ms 23.866 ms 23.661 ms

thunder.wiggy.net is my Linux router running 2.4.19-pre5-ac3-freeswan196
currently. The second hop is a normal sit tunnel and all the other
hops are native ipv6 using Cisco and Juniper routers as far as I know.
If you want I can get a detailed list of the routers and the IOS/JunOS
versions they are running.

Wichert.

--
Wichert Akkerman <[email protected]> http://www.wiggy.net/
A random hacker

2003-01-08 14:55:24

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Previously Wichert Akkerman wrote:
> traceroute to ipv6.lkml.org (2001:968:1::2) from
> 3ffe:8280:10:1d0:290:27ff:fe2d:968c, 30 hops max, 16 byte packets
> 1 thunder.wiggy.net (3ffe:8280:10:1d0:250:4ff:fe0b:dd79) 0.666 ms 0.22 ms 0.199 ms
> 2 xs4all-29.ipv6.xs4all.nl (3ffe:8280:0:2001::58) 27.568 ms 28.012 ms 30.177 ms
> 3 26.ge-0-2-0.xr1.pbw.xs4all.net (2001:888:0:3::1) 22.035 ms 19.528 ms 44.644 ms
> 4 0.ge-0-3-0.xr1.sara.xs4all.net (2001:888:2:1::1) 19.519 ms 19.002 ms 21.974 ms
> 5 fe-0-0-0.ams-core-01.network6.isp-services.nl (2001:7f8:1::a502:4875:1) 19.978 ms 30.278 ms 20.248 ms
> 6 2001:968::2 (2001:968::2) 24.246 ms 24.083 ms 22.918 ms
> 7 2001:968:1::2 (2001:968:1::2) 24.978 ms 23.866 ms 23.661 ms
>
> thunder.wiggy.net is my Linux router running 2.4.19-pre5-ac3-freeswan196
> currently. The second hop is a normal sit tunnel and all the other
> hops are native ipv6 using Cisco and Juniper routers as far as I know.

Slight correction to that: xs4all-29.ipv6.xs4all.nl is a FreeBSD-4.5
tunnel server. The toher xs4all.net hops are Junipers running JunOS 5.3
or 5.5.

Wichert.

--
Wichert Akkerman <[email protected]> http://www.wiggy.net/
A random hacker

2003-01-08 15:13:29

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Hi

On Wed, 8 Jan 2003, Wichert Akkerman wrote:

> tornado.wiggy.net is my client running Linux 2.4.19 (unpatched, UP machine),
> and 2001:968:1::2 is the icecast server running Linux 2.4.20-rc2-ac3 (SMP).
> If you want to test the stream yourself, please stream from
> http://ipv6.lkml.org:8000/difm .
>

I have no problem to stream from there. kernel-source-2.4.19 here. several
tunnels in the middle and different brand of routers...
anyway Im farly sure that the xmms patch is not the problem. We have been
testing it for more than 6 months now (for inclusion in debian, we are
not the upstream) and yes we hit a similar problems with one icecast
server but at that time we didn't care too much since it was basically at
the first tests round. I will see if i can get it up and running again
(it is not under my control) and investigate a bit more into it.

Thanks
Fabio

2003-01-08 16:30:55

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

> Previously Wichert Akkerman wrote:
> > traceroute to ipv6.lkml.org (2001:968:1::2) from
> > 3ffe:8280:10:1d0:290:27ff:fe2d:968c, 30 hops max, 16 byte packets
> > 1 thunder.wiggy.net (3ffe:8280:10:1d0:250:4ff:fe0b:dd79) 0.666 ms 0.22 ms 0.199 ms
> > 2 xs4all-29.ipv6.xs4all.nl (3ffe:8280:0:2001::58) 27.568 ms 28.012 ms 30.177 ms
> > 3 26.ge-0-2-0.xr1.pbw.xs4all.net (2001:888:0:3::1) 22.035 ms 19.528 ms 44.644 ms
> > 4 0.ge-0-3-0.xr1.sara.xs4all.net (2001:888:2:1::1) 19.519 ms 19.002 ms 21.974 ms
> > 5 fe-0-0-0.ams-core-01.network6.isp-services.nl (2001:7f8:1::a502:4875:1) 19.978 ms 30.278 ms 20.248 ms
> > 6 2001:968::2 (2001:968::2) 24.246 ms 24.083 ms 22.918 ms
> > 7 2001:968:1::2 (2001:968:1::2) 24.978 ms 23.866 ms 23.661 ms
> >
> > thunder.wiggy.net is my Linux router running 2.4.19-pre5-ac3-freeswan196
> > currently. The second hop is a normal sit tunnel and all the other
> > hops are native ipv6 using Cisco and Juniper routers as far as I know.
>
> Slight correction to that: xs4all-29.ipv6.xs4all.nl is a FreeBSD-4.5
> tunnel server. The toher xs4all.net hops are Junipers running JunOS 5.3
> or 5.5.
I had four contiguous listenings:
3 mins
10mins
19mins
13mins

When i increased the buffer in xmms i got better uninterrupted timings.
And survived data gaps better.

I seem to be getting better results than you, i think that it is not an
issue of ipv6 implementation but simply the case of time sensitive
traffic fighting with other Internet traffic over tunnels through ipv4
networks.

I do not know how many tunnels are in my path, i know that hop distance to
my tunnel is exactly 1 hop (ipv6 broker and ipv4 provider are the same)

If there is immense traffic at one of the routers (total traffic on an
interface) stream packets can be simply dropped if there are no queuing
disciplines that would take eg. flow control into account.

What do you think?

btw. what the hell is JunOs ?

Regards,
Maciej Soltysiak

2003-01-08 16:35:16

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Previously Maciej Soltysiak wrote:
> I do not know how many tunnels are in my path, i know that hop distance to
> my tunnel is exactly 1 hop (ipv6 broker and ipv4 provider are the same)

My tunnel provider is 5 hops away. To my knowledge non of the ipv4 or
ipv6 hops in the path are congested and no traffic shaping is done.

> If there is immense traffic at one of the routers (total traffic on an
> interface) stream packets can be simply dropped if there are no queuing
> disciplines that would take eg. flow control into account.

I'll ask the ISPs involved to check if this might be happening, but I
highly doubt it.

> btw. what the hell is JunOs ?

Juniper OS, running on Juniper routers.

Wichert.

--
Wichert Akkerman <[email protected]> http://www.wiggy.net/
A random hacker

2003-01-08 16:52:59

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Previously Maciej Soltysiak wrote:
> I seem to be getting better results than you, i think that it is not an
> issue of ipv6 implementation but simply the case of time sensitive
> traffic fighting with other Internet traffic over tunnels through ipv4
> networks.

Actually, I don't follow this. How could any kind of traffic shaping
result in my client not sending ACKs, which is what the tcpdump
seems to indicate? I can understand packets being dropped which
would result in retransmits, but that is not the case here.

Wichert.

(usual I'm-no-network-guru-and-might-be-misreading-things disclaimer here)

--
Wichert Akkerman <[email protected]> http://www.wiggy.net/
A random hacker

2003-01-08 17:35:07

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

> Actually, I don't follow this. How could any kind of traffic shaping
> result in my client not sending ACKs, which is what the tcpdump
> seems to indicate?
Well, i think i made a mistake, writing about routers dropping the
packets, it's not the case here, you are right.

Maciej

2003-01-08 17:56:57

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

I was able to reproduce the problem again. I have been using ethereal to
sniff instead of tcpdump and gave out some more info.

basically the icecast server at certain time (but i can't predict
exactly in which situations) just send a FIN, ACK packet to the client.
Basically to close the connection and after a few packets the client of
course answer.What is strange that in the meanwhile there are still 3/4
data packets coming from the server to the client.

Regarding the network side I noticed the following:

an average of 500ms to ping6 the server and 0 pkt loss
few seconds before the FIN, ACK (server->client) and for about 6 pkts the
average jumped to 2000ms

I suspect that this network flap made the server thinking about
<insert_here_whatever_term_is_more_appropriate> and decided to close
the connection.

The full ethereal dump is available at
http://www.fabbione.net/ice-xmms-ipv6.dump.bz2

but PLEASE note that it is a 10MB file and Im on a slow adsl line so be
"nice".

Fabio

PS Im afraid/happy that anyway the problem is not related to the kernel
version we are running.

--
vega:~# apt-get install life
Reading Package Lists... Done
Building Dependency Tree... Done
E: Couldn't find package life

2003-01-08 19:44:00

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Selective ACK is mandatory in IPv6 and uses a somewhat different algorithm,
so you shouldn't be seeing nearly as many ACKs as an IPv4 client would do
by default.

Andrew

--On Wednesday, January 08, 2003 18:01:39 +0100 Wichert Akkerman
<[email protected]> wrote:

> Previously Maciej Soltysiak wrote:
>> I seem to be getting better results than you, i think that it is not an
>> issue of ipv6 implementation but simply the case of time sensitive
>> traffic fighting with other Internet traffic over tunnels through ipv4
>> networks.
>
> Actually, I don't follow this. How could any kind of traffic shaping
> result in my client not sending ACKs, which is what the tcpdump
> seems to indicate? I can understand packets being dropped which
> would result in retransmits, but that is not the case here.
>
> Wichert.
>
> (usual I'm-no-network-guru-and-might-be-misreading-things disclaimer here)
>
> --
> Wichert Akkerman <[email protected]> http://www.wiggy.net/
> A random hacker
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
>

2003-01-08 19:51:40

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

--On Wednesday, January 08, 2003 19:05:36 +0100 Fabio Massimo Di Nitto
<[email protected]> wrote:

>
> I was able to reproduce the problem again. I have been using ethereal to
> sniff instead of tcpdump and gave out some more info.
>
> basically the icecast server at certain time (but i can't predict
> exactly in which situations) just send a FIN, ACK packet to the client.
> Basically to close the connection and after a few packets the client of
> course answer.What is strange that in the meanwhile there are still 3/4
> data packets coming from the server to the client.
>
> Regarding the network side I noticed the following:
>
> an average of 500ms to ping6 the server and 0 pkt loss
> few seconds before the FIN, ACK (server->client) and for about 6 pkts the
> average jumped to 2000ms
>
> I suspect that this network flap made the server thinking about
> <insert_here_whatever_term_is_more_appropriate> and decided to close
> the connection.

Probably on the server's side it got an ICMP Host Unreachable or two as
some router updated its tables, and decided to close the connection. The
FIN jumped the queue in one/several of the routers in the path, so it got
reordered relative to the data. This would imply that the router in
question had its route to you back by the time the FIN got there.

Wierd, but far from impossible.

>
> The full ethereal dump is available at
> http://www.fabbione.net/ice-xmms-ipv6.dump.bz2
>
> but PLEASE note that it is a 10MB file and Im on a slow adsl line so be
> "nice".

I think you provided enough info to tell what happened.

>
> Fabio
>
> PS Im afraid/happy that anyway the problem is not related to the kernel
> version we are running.

Doesn't look like a kernel problem.

Someone's got a dodgy link or a routing problem.

Andrew

2003-01-08 20:22:26

[permalink] [raw]

Subject: [OT] Re: ipv6 stack seems to forget to send ACKs

Definitly it is somehow unstable. Difficult to find the reason.
Anyway Im sure that there is asimmetric routing between me and
ipv6.lkml.org and I could see pkts coming back. It might not have been the
case for Wichert

Fabio

On Wed, 8 Jan 2003, Maciej Soltysiak wrote:

> > Probably on the server's side it got an ICMP Host Unreachable or two as
> > some router updated its tables, and decided to close the connection.
> Sounds reasonable to me. Could it mean that this router that we are
> talking about is simply slow or overloaded ?
>
> Maciej
>
>
>

2003-01-08 20:18:20

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

> Probably on the server's side it got an ICMP Host Unreachable or two as
> some router updated its tables, and decided to close the connection.
Sounds reasonable to me. Could it mean that this router that we are
talking about is simply slow or overloaded ?

Maciej

2003-01-08 20:31:01

[permalink] [raw]

Subject: Re: [OT] Re: ipv6 stack seems to forget to send ACKs

Probably not slow and/or overloaded. I'd think it's more likely that it
has a routing update problem or an unreliable link. But whatever, this
seemed to me to be a classic 'dodgy box in the middle' rather than an end
host problem.

Andrew

--On Wednesday, January 08, 2003 21:31:04 +0100 Fabio Massimo Di Nitto
<[email protected]> wrote:

>
> Definitly it is somehow unstable. Difficult to find the reason.
> Anyway Im sure that there is asimmetric routing between me and
> ipv6.lkml.org and I could see pkts coming back. It might not have been the
> case for Wichert
>
> Fabio
>
> On Wed, 8 Jan 2003, Maciej Soltysiak wrote:
>
>> > Probably on the server's side it got an ICMP Host Unreachable or two as
>> > some router updated its tables, and decided to close the connection.
>> Sounds reasonable to me. Could it mean that this router that we are
>> talking about is simply slow or overloaded ?
>>
>> Maciej
>>
>>
>>
>
>

2003-01-08 22:35:10

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Previously Andrew McGregor wrote:
> Probably on the server's side it got an ICMP Host Unreachable or two as
> some router updated its tables, and decided to close the connection.

The fact that this problem does not seem to occur when using a window
XP client seems to contradict the suggestions that it may be a router
problem.

Wichert.

--
Wichert Akkerman <[email protected]> http://www.wiggy.net/
A random hacker

2003-01-09 07:20:41

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

On Wed, 8 Jan 2003, Wichert Akkerman wrote:

> Previously Andrew McGregor wrote:
> > Probably on the server's side it got an ICMP Host Unreachable or two as
> > some router updated its tables, and decided to close the connection.
>
> The fact that this problem does not seem to occur when using a window
> XP client seems to contradict the suggestions that it may be a router
> problem.
>
> Wichert.
>
>

Is the WinXP client located in the same place where you are?

>From my side the ISP that is giving me problems is xs26.net
at 2 different points. One is flapping and one is the link between them
and another ISP (i can't even reach it now) where pkts get seriously
delayed (from 100ms to more than 350ms) probably due to a slow link.
But what is seriously annoying is that xs26.net keeps announcing a network
that it can't reach, fscking the entire routing.

Fabio

2003-01-09 08:24:35

by Herbert Xu

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Wichert Akkerman <[email protected]> wrote:
>
> 13:57:40.310471 tornado.wiggy.net.33035 > 2001:968:1::2.8000: . ack 9359225 win 32616 <nop,nop,timestamp 846112 369670744,nop,nop,sack sack 1 {9360653:9363289} >
> 13:57:40.325396 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9363289:9363509(220) ack 1 win 5712 <nop,nop,timestamp 369670750 846111> [class 0x2]
> 13:57:40.325447 tornado.wiggy.net.33035 > 2001:968:1::2.8000: . ack 9359225 win 32616 <nop,nop,timestamp 846113 369670744,nop,nop,sack sack 1 {9360653:9363509} >
> 13:57:40.568652 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9359225:9360433(1208) ack 1 win 5712 <nop,nop,timestamp 369670773 846113>
> 13:57:41.121608 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9359225:9360433(1208) ack 1 win 5712 <nop,nop,timestamp 369670829 846113>
> 13:57:42.242095 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9359225:9360433(1208) ack 1 win 5712 <nop,nop,timestamp 369670941 846113>
> 13:57:44.481379 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9359225:9360433(1208) ack 1 win 5712 <nop,nop,timestamp 369671165 846113>
> 13:57:48.963035 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9359225:9360433(1208) ack 1 win 5712 <nop,nop,timestamp 369671613 846113>

Verify the checksum of that packet, it's probably corrupt.
--
Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

2003-01-09 09:30:24

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Previously Fabio Massimo Di Nitto wrote:
> Is the WinXP client located in the same place where you are?

Yes. (Well, not my place but a friend who had both linux and XP did
that test).

Wichert.

--
Wichert Akkerman <[email protected]> http://www.wiggy.net/
A random hacker

2003-01-09 09:34:29

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

On Thu, 9 Jan 2003, Wichert Akkerman wrote:

> Previously Fabio Massimo Di Nitto wrote:
> > Is the WinXP client located in the same place where you are?
>
> Yes. (Well, not my place but a friend who had both linux and XP did
> that test).
>
> Wichert.
>
>

hmmmm strange because now I have forced the route to ipv6.lkml.org via
another ISP (bypassing xs26.net) and it is more than 3 hours that xmms is
streaming without interruptions.

Fabio

2003-01-09 10:24:08

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

> The fact that this problem does not seem to occur when using a window
> XP client seems to contradict the suggestions that it may be a router
> problem.
What other linux clients support streaming on ip6 ? patched mpg123 maybe?
What XP client are you using ?

Maybe it is a client issue, you say the client stops sending ACKs, maybe
the client code is buggy.

> Wichert.
Maciej.

2003-01-09 10:32:18

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Previously Maciej Soltysiak wrote:
> What other linux clients support streaming on ip6 ? patched mpg123 maybe?

The xmms patch can be adopted to work for mpg123 I suspect, I haven't
tried that.

> What XP client are you using ?

Iirc mediaplayer was used.

> Maybe it is a client issue, you say the client stops sending ACKs, maybe
> the client code is buggy.

I don't think a userspace tool can cause ACKs to stop being send.

Wichert.

--
Wichert Akkerman <[email protected]> http://www.wiggy.net/
A random hacker

2003-01-09 11:30:27

by Rogier Wolff

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

On Wed, Jan 08, 2003 at 02:08:50PM +0100, Wichert Akkerman wrote:
>
> 13:57:39.812123 tornado.wiggy.net.33035 > 2001:968:1::2.8000: . ack 9352713 win 32616 <nop,nop,timestamp 846062 369670698>
> 13:57:39.823581 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9352713:9353921(1208) ack 1 win 5712 <nop,nop,timestamp 369670698 846028> [class 0x2]
> 13:57:39.823636 tornado.wiggy.net.33035 > 2001:968:1::2.8000: . ack 9353921 win 32616 <nop,nop,timestamp 846063 369670698>

The packet is acked about 50 us after reception. Wicked!

> 13:57:39.835144 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9353921:9355129(1208) ack 1 win 5712 <nop,nop,timestamp 369670698 846028> [class 0x2]
> 13:57:39.835197 tornado.wiggy.net.33035 > 2001:968:1::2.8000: . ack 9355129 win 32616 <nop,nop,timestamp 846064 369670698>

Same here.

> 13:57:39.844277 2001:968:1::2.8000 > tornado.wiggy.net.33035: P 9355129:9355601(472) ack 1 win 5712 <nop,nop,timestamp 369670701 846062> [class 0x2]
> 13:57:39.844326 tornado.wiggy.net.33035 > 2001:968:1::2.8000: . ack 9355601 win 32616 <nop,nop,timestamp 846065 369670701>

> 13:57:40.221776 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9355601:9356809(1208) ack 1 win 5712 <nop,nop,timestamp 369670739 846065> [class 0x2]
> 13:57:40.221846 tornado.wiggy.net.33035 > 2001:968:1::2.8000: . ack 9356809 win 32616 <nop,nop,timestamp 846103 369670739>

> 13:57:40.233558 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9356809:9358017(1208) ack 1 win 5712 <nop,nop,timestamp 369670739 846065> [class 0x2]
> 13:57:40.233613 tornado.wiggy.net.33035 > 2001:968:1::2.8000: . ack 9358017 win 32616 <nop,nop,timestamp 846104 369670739>

> 13:57:40.245110 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9358017:9359225(1208) ack 1 win 5712 <nop,nop,timestamp 369670739 846065> [class 0x2]
> 13:57:40.245160 tornado.wiggy.net.33035 > 2001:968:1::2.8000: . ack 9359225 win 32616 <nop,nop,timestamp 846105 369670739>

same. same. same.

> 13:57:40.282351 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9359225:9360433(1208) ack 1 win 5712 <nop,nop,timestamp 369670744 846103>

But now: No ack! Funny.

> 13:57:40.284307 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9360433:9360653(220) ack 1 win 5712 <nop,nop,timestamp 369670744 846103>

Another packet, no ack!

> 13:57:40.297307 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9360653:9361861(1208) ack 1 win 5712 <nop,nop,timestamp 369670745 846104>
> 13:57:40.297376 tornado.wiggy.net.33035 > 2001:968:1::2.8000: . ack 9359225 win 32616 <nop,nop,timestamp 846111 369670744,nop,nop,sack sack 1 {9360653:9361861} >

Another packet, but this time it SACKs the just-recieved packet. It looks
as if the two packets inbetween somehow were not recognized as belonging
with this connection.

> 13:57:40.308222 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9362081:9363289(1208) ack 1 win 5712 <nop,nop,timestamp 369670745 846104> [class 0x2]
> 13:57:40.308271 tornado.wiggy.net.33035 > 2001:968:1::2.8000: . ack 9359225 win 32616 <nop,nop,timestamp 846112 369670744,nop,nop,sack sack 2 {9362081:9363289}{9360653:9361861} >

new packet, recognized ok, buffered, and acked!

> 13:57:40.310424 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9361861:9362081(220) ack 1 win 5712 <nop,nop,timestamp 369670745 846105>
> 13:57:40.310471 tornado.wiggy.net.33035 > 2001:968:1::2.8000: . ack 9359225 win 32616 <nop,nop,timestamp 846112 369670744,nop,nop,sack sack 1 {9360653:9363289} >
> 13:57:40.325396 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9363289:9363509(220) ack 1 win 5712 <nop,nop,timestamp 369670750 846111> [class 0x2]
> 13:57:40.325447 tornado.wiggy.net.33035 > 2001:968:1::2.8000: . ack 9359225 win 32616 <nop,nop,timestamp 846113 369670744,nop,nop,sack sack 1 {9360653:9363509} >

Two more packets, and still more hints towards the other machine that
we're missing 9359225-9360653

> 13:57:40.568652 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9359225:9360433(1208) ack 1 win 5712 <nop,nop,timestamp 369670773 846113>

So, it retransmits the first. but we don't see it as beloging to
this connection or something, so it gets ignored.

> 13:57:41.121608 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9359225:9360433(1208) ack 1 win 5712 <nop,nop,timestamp 369670829 846113>
> 13:57:42.242095 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9359225:9360433(1208) ack 1 win 5712 <nop,nop,timestamp 369670941 846113>
> 13:57:44.481379 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9359225:9360433(1208) ack 1 win 5712 <nop,nop,timestamp 369671165 846113>
> 13:57:48.963035 2001:968:1::2.8000 > tornado.wiggy.net.33035: . 9359225:9360433(1208) ack 1 win 5712 <nop,nop,timestamp 369671613 846113>

again, again, again.

It looks as if somehow those two packets 9359225:9360433 and 9360433:9360653 get
mangled in a way as to invalidate the checksum. This would cause "silent drop"
of these packets before they were acked....

Can you check the stats counters, to see if they are indeed dropped?

Roger.

--
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* The Worlds Ecosystem is a stable system. Stable systems may experience *
* excursions from the stable situation. We are currently in such an *
* excursion: The stable situation does not include humans. ***************

2003-01-09 11:47:03

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

--On Thursday, January 09, 2003 12:38:58 +0100 Rogier Wolff
<[email protected]> wrote:

> On Wed, Jan 08, 2003 at 02:08:50PM +0100, Wichert Akkerman wrote:
>>

Looked normal and then:

>
>> 13:57:40.282351 2001:968:1::2.8000 > tornado.wiggy.net.33035: .
>> 9359225:9360433(1208) ack 1 win 5712 <nop,nop,timestamp 369670744 846103>
>
> But now: No ack! Funny.

Might be SACK deciding not to...

>> 13:57:40.284307 2001:968:1::2.8000 > tornado.wiggy.net.33035: .
>> 9360433:9360653(220) ack 1 win 5712 <nop,nop,timestamp 369670744 846103>
>
> Another packet, no ack!
>
>> 13:57:40.297307 2001:968:1::2.8000 > tornado.wiggy.net.33035: .
>> 9360653:9361861(1208) ack 1 win 5712 <nop,nop,timestamp 369670745 846104>
>> 13:57:40.297376 tornado.wiggy.net.33035 > 2001:968:1::2.8000: . ack
>> 9359225 win 32616 <nop,nop,timestamp 846111 369670744,nop,nop,sack sack
>> 1 {9360653:9361861} >
>
> Another packet, but this time it SACKs the just-recieved packet. It looks
> as if the two packets inbetween somehow were not recognized as belonging
> with this connection.

or SACK forgot about them?

> Two more packets, and still more hints towards the other machine that
> we're missing 9359225-9360653
>
>> 13:57:40.568652 2001:968:1::2.8000 > tornado.wiggy.net.33035: .
>> 9359225:9360433(1208) ack 1 win 5712 <nop,nop,timestamp 369670773 846113>
>
> So, it retransmits the first. but we don't see it as beloging to
> this connection or something, so it gets ignored.

or we're waiting for the other one to ACK them both in one go?

> It looks as if somehow those two packets 9359225:9360433 and
> 9360433:9360653 get mangled in a way as to invalidate the checksum. This
> would cause "silent drop" of these packets before they were acked....

Could be data dependant, so there's a pattern in the packet contents that
causes this?

> Can you check the stats counters, to see if they are indeed dropped?
>
> Roger.

2003-01-09 15:32:50

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Previously Rogier Wolff wrote:
> Can you check the stats counters, to see if they are indeed dropped?

If you can tell me to which counter I am looking for and where I can
find that, sure.

Wichert.

--
Wichert Akkerman <[email protected]> http://www.wiggy.net/
A random hacker

2003-01-09 15:43:37

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Previously Rogier Wolff wrote:
> Can you check the stats counters, to see if they are indeed dropped?

It seems no packets are dropped:

Ip:
269716 total packets received
0 forwarded
0 incoming packets discarded
206853 incoming packets delivered
221800 requests sent out
75513 reassemblies required
12713 packets reassembled ok
10798 fragments created
Icmp:
0 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
0 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
Tcp:
666 active connections openings
0 passive connection openings
0 failed connection attempts
0 connection resets received
2 connections established
58949 segments received
65043 segments send out
0 segments retransmited
18 bad segments received.
82 resets sent
TcpExt:
ArpFilter: 0
7 TCP sockets finished time wait in fast timer
1091 delayed acks sent
2 delayed acks further delayed because of locked socket
6 packets directly queued to recvmsg prequeue.
6 packets directly received from prequeue
52427 packets header predicted
TCPPureAcks: 1259
TCPHPAcks: 10858
TCPRenoRecovery: 0
TCPSackRecovery: 0
TCPSACKReneging: 0
TCPFACKReorder: 0
TCPSACKReorder: 0
TCPRenoReorder: 0
TCPTSReorder: 0
TCPFullUndo: 0
TCPPartialUndo: 0
TCPDSACKUndo: 0
TCPLossUndo: 0
TCPLoss: 0
TCPLostRetransmit: 0
TCPRenoFailures: 0
TCPSackFailures: 0
TCPLossFailures: 0
TCPFastRetrans: 0
TCPForwardRetrans: 0
TCPSlowStartRetrans: 0
TCPTimeouts: 0
TCPRenoRecoveryFail: 0
TCPSackRecoveryFail: 0
TCPSchedulerFailed: 0
TCPRcvCollapsed: 0
TCPDSACKOldSent: 0
TCPDSACKOfoSent: 0
TCPDSACKRecv: 0
TCPDSACKOfoRecv: 0
TCPAbortOnSyn: 0
TCPAbortOnData: 24
TCPAbortOnClose: 67
TCPAbortOnMemory: 0
TCPAbortOnTimeout: 0
TCPAbortOnLinger: 0
TCPAbortFailed: 0
TCPMemoryPressures: 0

Wichert.

--
Wichert Akkerman <[email protected]> http://www.wiggy.net/
A random hacker

2003-01-09 15:54:40

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Previously Wichert Akkerman wrote:
> It seems no packets are dropped:

And now for ipv6 statistics from /proc/net/snmp6 (thanks Jamal):

Ip6InReceives 27011
Ip6InHdrErrors 0
Ip6InTooBigErrors 0
Ip6InNoRoutes 0
Ip6InAddrErrors 0
Ip6InUnknownProtos 0
Ip6InTruncatedPkts 0
Ip6InDiscards 0
Ip6InDelivers 26885
Ip6OutForwDatagrams 0
Ip6OutRequests 26848
Ip6OutDiscards 0
Ip6OutNoRoutes 0
Ip6ReasmTimeout 0
Ip6ReasmReqds 0
Ip6ReasmOKs 0
Ip6ReasmFails 0
Ip6FragOKs 0
Ip6FragFails 0
Ip6FragCreates 0
Ip6InMcastPkts 40
Ip6OutMcastPkts 0
Icmp6InMsgs 123
Icmp6InErrors 0
Icmp6InDestUnreachs 0
Icmp6InPktTooBigs 0
Icmp6InTimeExcds 0
Icmp6InParmProblems 0
Icmp6InEchos 0
Icmp6InEchoReplies 0
Icmp6InGroupMembQueries 0
Icmp6InGroupMembResponses 0
Icmp6InGroupMembReductions 0
Icmp6InRouterSolicits 0
Icmp6InRouterAdvertisements 40
Icmp6InNeighborSolicits 45
Icmp6InNeighborAdvertisements 38
Icmp6InRedirects 0
Icmp6OutMsgs 86
Icmp6OutDestUnreachs 0
Icmp6OutPktTooBigs 0
Icmp6OutTimeExcds 0
Icmp6OutParmProblems 0
Icmp6OutEchoReplies 0
Icmp6OutRouterSolicits 1
Icmp6OutNeighborSolicits 40
Icmp6OutNeighborAdvertisements 45
Icmp6OutRedirects 0
Icmp6OutGroupMembResponses 0
Icmp6OutGroupMembReductions 0
Udp6InDatagrams 0
Udp6NoPorts 0
Udp6InErrors 0
Udp6OutDatagrams 0

Wichert.

--
Wichert Akkerman <[email protected]> http://www.wiggy.net/
A random hacker

2003-01-09 22:03:50

by Mika Liljeberg

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Hi Wichert,

Looking at your trace it seems that the receiving machine is dropping
all packets that do not have traffic class set. Note that all segments
received with [class 0x2] get properly acked. The others probably don't
get to TCP at all. You might want to check your filters and QoS
policies.

BR,

MikaL

On Wed, 2003-01-08 at 19:01, Wichert Akkerman wrote:
> Previously Maciej Soltysiak wrote:
> > I seem to be getting better results than you, i think that it is not an
> > issue of ipv6 implementation but simply the case of time sensitive
> > traffic fighting with other Internet traffic over tunnels through ipv4
> > networks.
>
> Actually, I don't follow this. How could any kind of traffic shaping
> result in my client not sending ACKs, which is what the tcpdump
> seems to indicate? I can understand packets being dropped which
> would result in retransmits, but that is not the case here.
>
> Wichert.
>
> (usual I'm-no-network-guru-and-might-be-misreading-things disclaimer here)

2003-01-09 22:13:14

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

Previously Mika Liljeberg wrote:
> Looking at your trace it seems that the receiving machine is dropping
> all packets that do not have traffic class set. Note that all segments
> received with [class 0x2] get properly acked. The others probably don't
> get to TCP at all. You might want to check your filters and QoS
> policies.

I don't have any filters and QoS support isn't even enabled in this
kernel.

Wichert.

--
Wichert Akkerman <[email protected]> http://www.wiggy.net/
A random hacker

2003-01-09 22:44:04

by Bill Davidsen

[permalink] [raw]

Subject: Re: ipv6 stack seems to forget to send ACKs

On Thu, 9 Jan 2003, Andrew McGregor wrote:

> Probably on the server's side it got an ICMP Host Unreachable or two as
> some router updated its tables, and decided to close the connection. The
> FIN jumped the queue in one/several of the routers in the path, so it got
> reordered relative to the data. This would imply that the router in
> question had its route to you back by the time the FIN got there.
>
> Wierd, but far from impossible.

Nothing is impossible with routers, including sending ICMP packets using
different routing than TCP (and doing odd things with UDP as well).
Sometimes ICMP will be sent over a low bandwidth link with hopefully less
latency. The phrase "the less traveled way" comes to mind.

--
bill davidsen <[email protected]>
CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

2003-01-10 01:09:36