LinuxLists.cc - Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

Thanks for the empty posting. Please provide the content you
intended to post, and furthermore please post it to the network
developer mailing list, [email protected]

Thanks.

2005-09-01 22:49:24

by Jesper Juhl

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

On 9/2/05, David S. Miller <[email protected]> wrote:
>
> Thanks for the empty posting. Please provide the content you
> intended to post, and furthermore please post it to the network
> developer mailing list, [email protected]
>
Hmm, I see plenty of content in the post. Want me to farward you a
copy off list ?

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

2005-09-01 22:53:19

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

Hi David,

On Thu, 1 Sep 2005, David S. Miller wrote:

> Thanks for the empty posting. Please provide the content you
> intended to post, and furthermore please post it to the network
> developer mailing list, [email protected]

First of all, thanks for the reply (even to an empty posting :).

The posting wasn't actually empty, it was probably too long (94K according
to my sent-mail folder) and majordomo truncated it to zero. It has some
tcpdump snippets, that's what made it so long... unfortunately, they're
all necessary to understand the nature of the bug. I wasn't sure about
netdev, that's why I posted it only to linux-kernel and linux-net.

I can provide the full tcpdump out-of-band to interested people, since I
don't think I can get it past majordomo.

Here is the text of the message without the tcpdump inserts:

---------------------------------------------------------------------------
Hello,

I've been tracking down this bug for some time, and I'm fairly convinced
at this point that it's a kernel bug.

Under certain conditions, the TCP stack starts shrinking the TCP window
down to some ridiculously low values (hundreds of bytes, as low as 181)
and never recovers. The certain conditions I mentioned are not well
understood at this point, but they include a long-lived connection with a
very one-sided, fluctuating traffic flowing through it.

So far I've been able to reproduce it on plain-vanilla 2.4.9, 2.4.11.9,
and 2.4.12.2, as well as on the RHEL3 kernels 2.4.21-20 and 2.4.21-31. The
hardware is dual Opteron 250, running both 32- and 64-bit SMP kernels
(seems to make no difference). I've also seen the bug occur on a single
Athlon XP running 2.6.11.9 UP.

The bug occurs with all sysctl settings at their default values. I've
tried enabling and disabling pretty much all the tcp-related sysctl's in
/proc/sys/net/ipv4, to no visible improvement.

Here are a few tcpdump snippets of a TCP connection exhibiting the bug
(the complete tcpdump is available upon request, but it's very large).
10.2.20.246 is the data receiver and is the box exhibiting the bug (I'm
not sure what 10.2.224.182 is running, I don't have access to it). The
data being sent through is real-time financial data; the session begins by
catching up (at line speed) to present time, then continues to receive
real-time data as it is being generated. For what it's worth, we've never
been seen the bug occur while the session is still catching up (and
receiving a few large packets at a time); it always seems to happen while
receiving real-time data (many small packets, variably interspaced).

[I apologize for the amount of tcpdump data, but it's the only way to show
the bug in action.]

[tcpdump output removed]

The connection is established and the receiver's TCP window quickly ramps
up to 8192.

[tcpdump output removed]

Shortly thereafter the TCP window increases further to 16534. It remains
around 16534 for the next 5 minutes or so.

[tcpdump output removed]

A few minutes later it has finally caught up to present time and it starts
receiving smaller packets containing real-time data. The TCP window is
still 16534 at this point.

[tcpdump output removed]

This is where things start going bad. The window starts shrinking from
15340 all the way down to 2355 over the course of 0.3 seconds. Notice the
many duplicate acks that serve no purpose (there are no lost packets and
the tcpdump is taken on the receiver so there is no packets/acks crossed
in flight).

[tcpdump output removed]

Five minutes later the TCP window is still at 2355, having never
recovered. The window is so small that the available bandwidth for this
connection is too small to keep up with the real-time data so it is
falling behind, hence large packets are again being used. The application
processing the data (Java-based) is mostly idle at this point, and netstat
shows its recv queue to be empty. There is no apparent reason why the
kernel shouldn't enlarge the window.

In fact, if I let it continue, it eventually shrinks the window even
further (by 18:19:29, the time I'm writing this email, it's gone all the
way down to 1373). As I mentioned earlier, I've seen it go as low as 181.

We are kind of stumped at this point, and it's proving to be a
show-stopping bug for our purposes, especially over WAN links that have
higher latency (for obvious reasons). Any kind of assistance would be
greatly appreciated.

Thanks,
-Ion

2005-09-01 22:53:25

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

From: Jesper Juhl <[email protected]>
Date: Fri, 2 Sep 2005 00:49:20 +0200

> Hmm, I see plenty of content in the post. Want me to farward you a
> copy off list ?

Please do, I didn't see anything.

It still needs to be reposted to [email protected]
anyways :)

2005-09-01 23:37:07

by Jesper Juhl

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

On 9/2/05, Ion Badulescu <[email protected]> wrote:
> Hi David,
>
> On Thu, 1 Sep 2005, David S. Miller wrote:
>
> > Thanks for the empty posting. Please provide the content you
> > intended to post, and furthermore please post it to the network
> > developer mailing list, [email protected]
>
> First of all, thanks for the reply (even to an empty posting :).
>
> The posting wasn't actually empty, it was probably too long (94K according

Two solutions commonly applied to that problem :

- put the big file(s) online somewhere and include an URL in the email
- compress the file(s) and attach the compressed files to the email

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

2005-09-02 02:51:56

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

On Sep 1, 2005, at 6:53 PM, Ion Badulescu wrote:
>
> A few minutes later it has finally caught up to present time and it
> starts receiving smaller packets containing real-time data. The TCP
> window is still 16534 at this point.
>
> [tcpdump output removed]
>
> This is where things start going bad. The window starts shrinking from
> 15340 all the way down to 2355 over the course of 0.3 seconds. Notice
> the many duplicate acks that serve no purpose (there are no lost
> packets and the tcpdump is taken on the receiver so there is no
> packets/acks crossed in flight).

I have an idea why this is going on. Packets are pre-allocated by the
driver to be a max packet size, so when you send small packets, it
wastes a lot of memory. Currently Linux uses the packets at the
beginning of a connection to make a guess at how best to advertise its
window so as not to overflow the socket's memory bounds. Since you
start out with big segments then go to small ones, this is defeating
that mechanism. It's actually documented in the comments in
tcp_input.c. :)

* The scheme does not work when sender sends good segments opening
* window and then starts to feed us spagetti. But it should work
* in common situations. Otherwise, we have to rely on queue collapsing.

If you overflow the socket's memory bound, it ends up calling
tcp_clamp_window(). (I'm not sure this is really the right thing to do
here before trying to collapse the queue.) If the receiving
application doesn't fall too far behind, it might help you to set a
much larger receiver buffer.

-John

2005-09-02 04:52:03

by Noritoshi Demizu

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

Below may not be directly related to the cause of the problem.
But I think some window sizes in your mail need to be re-evaluated.

> 11:29:54.961998 10.2.20.246.33060 > 10.2.224.182.8700: S 1972343059:1972343059(0) win 5840 <mss 1460,sackOK,timestamp 225781001 0,nop,wscale 2> (DF)
> 11:29:54.983334 10.2.224.182.8700 > 10.2.20.246.33060: S 2770690746:2770690746(0) ack 1972343060 win 33304 <nop,nop,timestamp 99687881 225781001,mss 1460,nop,wscale 1,nop,nop,sackOK> (DF)

Since the TCP Window Scale options are exchanged,
the window size field contains shifted value except SYNs.

> 11:29:55.216634 10.2.20.246.33060 > 10.2.224.182.8700: . ack 14307 win 8192 <nop,nop,timestamp 225781255 99687903> (DF)
>
> The connection is established and the receiver's TCP window quickly ramps
> up to 8192.

I think the real window size here is 8192 << 2 = 32768.

> 11:29:55.707823 10.2.20.246.33060 > 10.2.224.182.8700: . ack 274527 win 16534 <nop,nop,timestamp 225781747 99687948> (DF)
>
> Shortly thereafter the TCP window increases further to 16534. It remains
> around 16534 for the next 5 minutes or so.

I think the real window size here is 16534 << 2 = 66136.

> 11:31:09.345250 10.2.20.246.33060 > 10.2.224.182.8700: . ack 38676603 win 16534 <nop,nop,timestamp 225855408 99695308> (DF)
>
> A few minutes later it has finally caught up to present time and it starts
> receiving smaller packets containing real-time data. The TCP window is
> still 16534 at this point.

I think the real window size here is 16534 << 2 = 66136.

> 11:34:54.337167 10.2.20.246.33060 > 10.2.224.182.8700: . ack 84402527 win 15340 <nop,nop,timestamp 226080473 99717814> (DF)
(sniop)
> 11:34:54.628497 10.2.20.246.33060 > 10.2.224.182.8700: . ack 84458619 win 2355 <nop,nop,timestamp 226080737 99717845> (DF)
>
> This is where things start going bad. The window starts shrinking from
> 15340 all the way down to 2355 over the course of 0.3 seconds.

I think the real window sizes are 15340 << 2 = 61360 and 2355 << 2 = 9420.
Hence, the right edges can be calculated as the following:

84402527 + 15340 << 2 = 84463887
84458619 + 2355 << 2 = 84468039

So, the window is not shrinked between those two lines.
Sorry, I have not checked other lines.

> 11:40:08.279678 10.2.20.246.33060 > 10.2.224.182.8700: . ack 134973263 win 2355 <nop,nop,timestamp 226394517 99749214> (DF)
>
> Five minutes later the TCP window is still at 2355, having never recovered.

I think the real window size here is 2355 << 2 = 9420.

> As I mentioned earlier, I've seen it go as low as 181.

I think the real window size is 181 << 2 = 724.

Regards,
Noritoshi Demizu

2005-09-02 05:19:18

by Stephen Hemminger

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

On Fri, 02 Sep 2005 13:51:38 +0900 (JST)
Noritoshi Demizu <[email protected]> wrote:

> Below may not be directly related to the cause of the problem.
> But I think some window sizes in your mail need to be re-evaluated.
>
> > 11:29:54.961998 10.2.20.246.33060 > 10.2.224.182.8700: S 1972343059:1972343059(0) win 5840 <mss 1460,sackOK,timestamp 225781001 0,nop,wscale 2> (DF)
> > 11:29:54.983334 10.2.224.182.8700 > 10.2.20.246.33060: S 2770690746:2770690746(0) ack 1972343060 win 33304 <nop,nop,timestamp 99687881 225781001,mss 1460,nop,wscale 1,nop,nop,sackOK> (DF)
>
> Since the TCP Window Scale options are exchanged,
> the window size field contains shifted value except SYNs.

Be careful, tcpdump may be tracking the window scale and reporting
scaled values. Seems unlikely since with a window scale of 2, and odd
window size would be impossible. Also, is there by any chance a busted
firewall in the way (like OpenBSD) that corrupts window scaling.

2005-09-02 05:45:40

by Noritoshi Demizu

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

> > Since the TCP Window Scale options are exchanged,
> > the window size field contains shifted value except SYNs.
>
> Be careful, tcpdump may be tracking the window scale and reporting
> scaled values. Seems unlikely since with a window scale of 2, and odd
> window size would be impossible.

I am sorry if tcpdump tracks the window scale option. But I think it
is unlikely, because there are odd window sizes, as you pointed out.

By the way, if tcpdump does not track the window scale option, the right
edge (ack + real win) does not change between the following two ACKs.

> 11:34:54.337167 10.2.20.246.33060 > 10.2.224.182.8700: . ack 84402527 win 15340 <nop,nop,timestamp 226080473 99717814> (DF)
(259 ACKs are omitted here)
> 11:34:54.611769 10.2.20.246.33060 > 10.2.224.182.8700: . ack 84454467 win 2355 <nop,nop,timestamp 226080721 99717841> (DF)

The first line is the 37th ACK and the second line is the 295th ACK.

ACK#37: ack=84402527 win=15340 right_edge=84463887 (= ack + win * 4)
ACK#295: ack=84454467 win=2355 right_edge=84463887 (= ack + win * 4)

And all ACKs later than ACK#295 has win=2355 (2355*4=9420).

This may be a hint. But, sorry, I do not know the internal of Linux TCP.

Regards,
Noritoshi Demizu

2005-09-02 06:11:45

by Noritoshi Demizu

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

> By the way, if tcpdump does not track the window scale option, the right
> edge (ack + real win) does not change between the following two ACKs.
>
> > 11:34:54.337167 10.2.20.246.33060 > 10.2.224.182.8700: . ack 84402527 win 15340 <nop,nop,timestamp 226080473 99717814> (DF)
> (259 ACKs are omitted here)
> > 11:34:54.611769 10.2.20.246.33060 > 10.2.224.182.8700: . ack 84454467 win 2355 <nop,nop,timestamp 226080721 99717841> (DF)
>
> The first line is the 37th ACK and the second line is the 295th ACK.
>
> ACK#37: ack=84402527 win=15340 right_edge=84463887 (= ack + win * 4)
> ACK#295: ack=84454467 win=2355 right_edge=84463887 (= ack + win * 4)
>
> And all ACKs later than ACK#295 has win=2355 (2355*4=9420).
>
> This may be a hint. But, sorry, I do not know the internal of Linux TCP.

I think there is a possibility that some middle-box does something,
for example, some middle-box between the two machines does kinda
traffic-shaping by tweaking the TCP window size field.

Regards,
Noritoshi Demizu

2005-09-02 06:28:26

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

From: John Heffner <[email protected]>
Date: Thu, 1 Sep 2005 22:51:48 -0400

> I have an idea why this is going on. Packets are pre-allocated by the
> driver to be a max packet size, so when you send small packets, it
> wastes a lot of memory. Currently Linux uses the packets at the
> beginning of a connection to make a guess at how best to advertise its
> window so as not to overflow the socket's memory bounds. Since you
> start out with big segments then go to small ones, this is defeating
> that mechanism. It's actually documented in the comments in
> tcp_input.c. :)
>
> * The scheme does not work when sender sends good segments opening
> * window and then starts to feed us spagetti. But it should work
> * in common situations. Otherwise, we have to rely on queue collapsing.

That's a strong possibility, good catch John.

Although, I'm still not ruling out some box in the middle
even though I consider it less likely than your theory.

So you're suggesting that tcp_prune_queue() should do the:

if (atomic_read(&sk->sk_rmem_alloc) >= sk->sk_rcvbuf)
tcp_clamp_window(sk, tp);

check after attempting to collapse the queue.

But, that window clamping should fix the problem, as we recalculate
the window to advertise.

2005-09-02 12:11:25

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

On Fri, 2 Sep 2005, Noritoshi Demizu wrote:

>> By the way, if tcpdump does not track the window scale option, the right
>> edge (ack + real win) does not change between the following two ACKs.
>>
>>> 11:34:54.337167 10.2.20.246.33060 > 10.2.224.182.8700: . ack 84402527 win 15340 <nop,nop,timestamp 226080473 99717814> (DF)
>> (259 ACKs are omitted here)
>>> 11:34:54.611769 10.2.20.246.33060 > 10.2.224.182.8700: . ack 84454467 win 2355 <nop,nop,timestamp 226080721 99717841> (DF)
>>
>> The first line is the 37th ACK and the second line is the 295th ACK.
>>
>> ACK#37: ack=84402527 win=15340 right_edge=84463887 (= ack + win * 4)
>> ACK#295: ack=84454467 win=2355 right_edge=84463887 (= ack + win * 4)
>>
>> And all ACKs later than ACK#295 has win=2355 (2355*4=9420).
>>
>> This may be a hint. But, sorry, I do not know the internal of Linux TCP.

Oh, it's absolutely possible (even likely) that the application was slow
between 11:34:54.337167 and 11:34:54.611769 and data kept accumulating in
the socket buffer. The real problem is not the shrinking of the window,
but the fact that it never increases back to normal once the socket buffer
is emptied.

> I think there is a possibility that some middle-box does something,
> for example, some middle-box between the two machines does kinda
> traffic-shaping by tweaking the TCP window size field.

Not really: the tcpdump is taken on the very box that generates the acks
with the shrinking window, so it can't possibly be affected by any shaper.
Unless the shaper is the Linux kernel itself...

-Ion

2005-09-02 13:01:42

by Guillaume Autran

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

I experienced the very same problem but with window size going all the
way down to just a few bytes (14 bytes). dump files available upon
requests :)
Ion, how were you able to reproduce the issue ? Can the same type of
traffice always reproduce the issue or is it more intermittent ?

Best regards,
Guillaume.

John Heffner wrote:

> On Sep 1, 2005, at 6:53 PM, Ion Badulescu wrote:
>
>>
>> A few minutes later it has finally caught up to present time and it
>> starts receiving smaller packets containing real-time data. The TCP
>> window is still 16534 at this point.
>>
>> [tcpdump output removed]
>>
>> This is where things start going bad. The window starts shrinking
>> from 15340 all the way down to 2355 over the course of 0.3 seconds.
>> Notice the many duplicate acks that serve no purpose (there are no
>> lost packets and the tcpdump is taken on the receiver so there is no
>> packets/acks crossed in flight).
>
>
> I have an idea why this is going on. Packets are pre-allocated by the
> driver to be a max packet size, so when you send small packets, it
> wastes a lot of memory. Currently Linux uses the packets at the
> beginning of a connection to make a guess at how best to advertise its
> window so as not to overflow the socket's memory bounds. Since you
> start out with big segments then go to small ones, this is defeating
> that mechanism. It's actually documented in the comments in
> tcp_input.c. :)
>
> * The scheme does not work when sender sends good segments opening
> * window and then starts to feed us spagetti. But it should work
> * in common situations. Otherwise, we have to rely on queue collapsing.
>
> If you overflow the socket's memory bound, it ends up calling
> tcp_clamp_window(). (I'm not sure this is really the right thing to
> do here before trying to collapse the queue.) If the receiving
> application doesn't fall too far behind, it might help you to set a
> much larger receiver buffer.
>
> -John
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-net" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>

--
=======================================
Guillaume Autran
Senior Software Engineer
MRV Communications, Inc.
Tel: (978) 952-4932 office
E-mail: [email protected]
=======================================

2005-09-02 13:48:47

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

Hello!

> If you overflow the socket's memory bound, it ends up calling
> tcp_clamp_window(). (I'm not sure this is really the right thing to do
> here before trying to collapse the queue.)

Collapsing is too expensive procedure, it is rather an emergency measure.
So, tcp collapses queue, when it is necessary, but it must reduce window
as well.

Alexey

2005-09-02 13:48:41

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

On Fri, 2 Sep 2005, Guillaume Autran wrote:

> I experienced the very same problem but with window size going all the way
> down to just a few bytes (14 bytes). dump files available upon requests :)
> Ion, how were you able to reproduce the issue ? Can the same type of traffice
> always reproduce the issue or is it more intermittent ?

I have no problem whatsoever reproducing it, at least with the kind of
traffic I described. I had 4 flows like that running yesterday, and all 4
had TCP window sizes smaller than 500 bytes on the receiver by mid-day.

-Ion

2005-09-02 13:53:19

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

Hello!

> I experienced the very same problem but with window size going all the
> way down to just a few bytes (14 bytes). dump files available upon
> requests :)

I do request.

TCP is not allowed to reduce window to a value less than 2*MSS no matter
how hard network device or peer try to confuse it. :-)

Alexey

2005-09-02 14:05:28

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

Hi David,

On Thu, 1 Sep 2005, David S. Miller wrote:

> From: John Heffner <[email protected]>
> Date: Thu, 1 Sep 2005 22:51:48 -0400
>
>> I have an idea why this is going on. Packets are pre-allocated by the
>> driver to be a max packet size, so when you send small packets, it
>> wastes a lot of memory. Currently Linux uses the packets at the
>> beginning of a connection to make a guess at how best to advertise its
>> window so as not to overflow the socket's memory bounds. Since you
>> start out with big segments then go to small ones, this is defeating
>> that mechanism. It's actually documented in the comments in
>> tcp_input.c. :)
>>
>> * The scheme does not work when sender sends good segments opening
>> * window and then starts to feed us spagetti. But it should work
>> * in common situations. Otherwise, we have to rely on queue collapsing.
>
> That's a strong possibility, good catch John.

That's possible, but see below.

> Although, I'm still not ruling out some box in the middle
> even though I consider it less likely than your theory.

There is no funky box in the middle, that much I can guarantee you.

I said yesterday that I don't have access to the sender. While that's true
for the flow I had captured in those dumps, I saw the same phenomenon
occur between two boxes I control fully. The sender is running Windows
2000, and is separated from the receiver by a single Catalyst 6500
switch/router (they are on different VLAN's) which doesn't do anything
fancy (I control the switch as well).

This particular Win2k sender sends _only_ real-time data, it's not capable
of rewinding. So it's always sending small packets, from start to finish,
yet the problem still occurs.

Note that even real-time data can end up generating a stream of full-size
packets occassionally. It's just very unlikely they would occur at the
start of the flow, as market data is very thin in the pre-market open hours.

> So you're suggesting that tcp_prune_queue() should do the:
>
> if (atomic_read(&sk->sk_rmem_alloc) >= sk->sk_rcvbuf)
> tcp_clamp_window(sk, tp);
>
> check after attempting to collapse the queue.
>
> But, that window clamping should fix the problem, as we recalculate
> the window to advertise.

Patches for testing are very much welcome...

Thanks,
-Ion

2005-09-02 14:10:23

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

On Sep 2, 2005, at 10:05 AM, [email protected] wrote:

> This particular Win2k sender sends _only_ real-time data, it's not
> capable of rewinding. So it's always sending small packets, from start
> to finish, yet the problem still occurs.
>
> Note that even real-time data can end up generating a stream of
> full-size packets occassionally. It's just very unlikely they would
> occur at the start of the flow, as market data is very thin in the
> pre-market open hours.

The rcv_ssthresh growth can actually take place anywhere in the flow,
not just at the beginning.

>> But, that window clamping should fix the problem, as we recalculate
>> the window to advertise.
>
> Patches for testing are very much welcome...

Have you tried increasing the size of the receive buffer yet?

-John

2005-09-02 14:11:34

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

On Sep 2, 2005, at 9:52 AM, Alexey Kuznetsov wrote:

> Hello!
>
>> I experienced the very same problem but with window size going all the
>> way down to just a few bytes (14 bytes). dump files available upon
>> requests :)
>
> I do request.
>
> TCP is not allowed to reduce window to a value less than 2*MSS no
> matter
> how hard network device or peer try to confuse it. :-)

You're right, that doesn't make sense...

2005-09-02 14:17:11

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

On Sep 2, 2005, at 9:48 AM, Alexey Kuznetsov wrote:

> Hello!
>
>> If you overflow the socket's memory bound, it ends up calling
>> tcp_clamp_window(). (I'm not sure this is really the right thing to
>> do
>> here before trying to collapse the queue.)
>
> Collapsing is too expensive procedure, it is rather an emergency
> measure.
> So, tcp collapses queue, when it is necessary, but it must reduce
> window
> as well.

Right.

I wonder if clamping the window though is too harsh. Maybe just
setting the rcv_ssthresh down is better? Why the distinction between
in-order and out-of-order data? Because you expect in-order data to be
a persistent case?

-John

2005-09-02 14:34:09

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

On Fri, 2 Sep 2005, John Heffner wrote:

> Have you tried increasing the size of the receive buffer yet?

Actually, I just did. I changed rmem_max and rmem_default to 4MB and
tcp_rmem to "64k 4MB 4MB". It did seem to help, but I'm wondering if
that's simply because it has a _lot_ of memory now to leak before it
starts eating up into the window size.

I also ran into some very strange packet loss problems that weren't
occurring yesterday; they only started occurring after I increased the
buffer size. Most strange. If they happen again, I'll make sure I capture
the flow to analyze it.

Anyway, that was just to see if I can do anything at all to mitigate the
problem. I'll try again with smaller buffers (4k 128k 256k) and see what
happens.

Thanks,
-Ion

2005-09-02 14:48:09

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

On Sep 2, 2005, at 10:33 AM, [email protected] wrote:

> On Fri, 2 Sep 2005, John Heffner wrote:
>
>> Have you tried increasing the size of the receive buffer yet?
>
> Actually, I just did. I changed rmem_max and rmem_default to 4MB and
> tcp_rmem to "64k 4MB 4MB". It did seem to help, but I'm wondering if
> that's simply because it has a _lot_ of memory now to leak before it
> starts eating up into the window size.

If it is window clamping, then you should be asymptotically approaching
a ratio between receive buffer and window that corresponds (with a
fudge factor) to the ratio between TCP segment data size and allocated
packet size. If you make the receive buffer large enough, then the
clamped window should still end up big enough. Also, since you have
"real time" data, a larger receive buffer should probably be adequate
to eliminate this problem, since it only occurs when the receiving
application falls behind for a while, and a bigger receive buffer
allows it to fall behind more without triggering the window clamping.

-John

2005-09-02 15:12:09

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

Hello!

> I wonder if clamping the window though is too harsh. Maybe just
> setting the rcv_ssthresh down is better?

It is too harsh. This was invented before we learned how to collapse
received data, that time tiny segments were fatal and clamping was
the last weapon against misbehaving connections.

It can be removed.

Actually, right solution would be an attempt to calculate ratio
window/rcvbuf dynamically. It looked quite tricky, so it was not done.
Instead it is controlled with static sysctl sysctl_tcp_adv_win_scale.
It does not work sometimes f.e. when a device has larger link level overhead.
I think, this should be reconsidered.

> Why the distinction between
> in-order and out-of-order data? Because you expect in-order data to be
> a persistent case?

Overflow in in-order data is hard, we cannot drop data. Also, it means
that receiving application cannot hold to receive rate and we can shrink
window.

Out-of-order data are different: we can drop the segments if we are
in serious troubles and overflow there can be cured by expansion of window
to allow fast retransmit.

Alexey

2005-09-02 15:44:01

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

On Fri, 2 Sep 2005, John Heffner wrote:

> If it is window clamping, then you should be asymptotically approaching a
> ratio between receive buffer and window that corresponds (with a fudge
> factor) to the ratio between TCP segment data size and allocated packet size.
> If you make the receive buffer large enough, then the clamped window should
> still end up big enough.

For what it's worth, running with a 512k receive buffer still caused the
clamping to occur, though it took longer than with the normal buffer size.
The window went down from a maximum of 12291 (times 2^4 due to window
scaling) to 3190 currently. That's still enough for our purposes, but I'll
keep monitoring it to see if it shrinks any further. It could be a viable
work-around for the time being.

Is this a bug, though, or a feature? :)

> Also, since you have "real time" data, a larger
> receive buffer should probably be adequate to eliminate this problem, since
> it only occurs when the receiving application falls behind for a while, and a
> bigger receive buffer allows it to fall behind more without triggering the
> window clamping.

Correct. I noticed too while experimenting that the clamping never occurs
if the application is fast enough to keep the socket buffer empty. It's
when data is allowed to accumulate in the buffer that the window shrinks,
and then it never grows back, as if a portion of the buffer got lost
permanently.

-Ion

2005-09-02 18:37:30

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

Hello!

> This is where things start going bad. The window starts shrinking from
> 15340 all the way down to 2355 over the course of 0.3 seconds. Notice the
> many duplicate acks that serve no purpose

These are not duplicate, TCP_NODELAY sender just starts flooding
tiny segments, and those are normal ACKs acking those segments, note
ACK field is not the same.

> Five minutes later the TCP window is still at 2355,
....
> We are kind of stumped at this point, and it's proving to be a
> show-stopping bug for our purposes, especially over WAN links that have
> higher latency (for obvious reasons). Any kind of assistance would be
> greatly appreciated.

I still do not know how the value of 184 is possible in your case,
I would expect 730 as an absolute possible minumum. I see 9420 (2355*4).
Anyway, ignoring this puzzle, the following patch for 2.4 should help.

--- net/ipv4/tcp_input.c.orig 2003-02-20 20:38:39.000000000 +0300
+++ net/ipv4/tcp_input.c 2005-09-02 22:28:00.845952888 +0400
@@ -343,8 +343,6 @@
app_win -= tp->ack.rcv_mss;
app_win = max(app_win, 2U*tp->advmss);

- if (!ofo_win)
- tp->window_clamp = min(tp->window_clamp, app_win);
tp->rcv_ssthresh = min(tp->window_clamp, 2U*tp->advmss);
}
}

2005-09-02 20:58:56

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

Hi Alexey,

On Fri, 2 Sep 2005, Alexey Kuznetsov wrote:

>> This is where things start going bad. The window starts shrinking from
>> 15340 all the way down to 2355 over the course of 0.3 seconds. Notice the
>> many duplicate acks that serve no purpose
>
> These are not duplicate, TCP_NODELAY sender just starts flooding
> tiny segments, and those are normal ACKs acking those segments, note
> ACK field is not the same.

Well, take a look at the double acks for 84439343, 84440447 and 84441059,
they seem pretty much identical to me.

> I still do not know how the value of 184 is possible in your case,
> I would expect 730 as an absolute possible minumum. I see 9420 (2355*4).

The numbers I mentioned are straight from the tcpdump and are not scaled,
so they need to be multiplied by 4. But even 9420, combined with a RTT of
20ms, results in a total usable bandwidth of about 3.75 Mbps, not enough
for this real-time stream at peak times.

Besides, it often gets even worse than 2355, all it takes is a few
application slowdowns.

> Anyway, ignoring this puzzle, the following patch for 2.4 should help.
>
>
> --- net/ipv4/tcp_input.c.orig 2003-02-20 20:38:39.000000000 +0300
> +++ net/ipv4/tcp_input.c 2005-09-02 22:28:00.845952888 +0400
> @@ -343,8 +343,6 @@
> app_win -= tp->ack.rcv_mss;
> app_win = max(app_win, 2U*tp->advmss);
>
> - if (!ofo_win)
> - tp->window_clamp = min(tp->window_clamp, app_win);
> tp->rcv_ssthresh = min(tp->window_clamp, 2U*tp->advmss);
> }
> }

That makes perfect sense...

I'll test it out on Tuesday, when I can connect again to the real-time
streams that we use.

Thanks a lot!
-Ion

2005-09-02 21:18:31

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

Hello!

> Well, take a look at the double acks for 84439343, 84440447 and 84441059,
> they seem pretty much identical to me.

It is just a little tcpdump glitch.

19:34:54.532271 < 10.2.20.246.33060 > 65.171.224.182.8700: . 44:44(0) ack 84439343 win 24544 <nop,nop,timestamp 226080638 99717832> (DF) (ttl 64, id 60946)
19:34:54.532432 < 10.2.20.246.33060 > 65.171.224.182.8700: . 44:44(0) ack 84439343 win 24544 <nop,nop,timestamp 226080638 99717832> (DF) (ttl 64, id 60946)

It is one ACK (look at IP ID), shown twice. This happens sometimes
with our packet socket.

> >I still do not know how the value of 184 is possible in your case,
> >I would expect 730 as an absolute possible minumum. I see 9420 (2355*4).
>
> The numbers I mentioned are straight from the tcpdump and are not scaled,

I understood. I expect when 184*4, when you said 184. But minimum is
still 730 (unscaled 1460*2). If you really saw values lower than 730
(unscaled 1460*2), there is another more severe problem and the suggested
patch will not solve it.

Alexey

2005-09-02 23:09:42

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

Hi Alexey,

On Sat, 3 Sep 2005, Alexey Kuznetsov wrote:

>> Well, take a look at the double acks for 84439343, 84440447 and 84441059,
>> they seem pretty much identical to me.
>
> It is just a little tcpdump glitch.
>
> 19:34:54.532271 < 10.2.20.246.33060 > 65.171.224.182.8700: . 44:44(0) ack 84439343 win 24544 <nop,nop,timestamp 226080638 99717832> (DF) (ttl 64, id 60946)
> 19:34:54.532432 < 10.2.20.246.33060 > 65.171.224.182.8700: . 44:44(0) ack 84439343 win 24544 <nop,nop,timestamp 226080638 99717832> (DF) (ttl 64, id 60946)
>
> It is one ACK (look at IP ID), shown twice. This happens sometimes
> with our packet socket.

Ahh... ack. :) That explains it.

> I understood. I expect when 184*4, when you said 184. But minimum is
> still 730 (unscaled 1460*2). If you really saw values lower than 730
> (unscaled 1460*2), there is another more severe problem and the suggested
> patch will not solve it.

I really did see very small values. This one is plucked from one of
today's streams, after a full day's worth of data had passed through it:

19:03:19.659454 10.1.12.11.8001 > 10.2.10.212.56690: P 3:6(3) ack 1 win 65529 <nop,nop,timestamp 27146219 3617561665> (DF)
19:03:19.659462 10.2.10.212.56690 > 10.1.12.11.8001: . ack 6 win 181 <nop,nop,timestamp 3617562713 27146219> (DF)
19:03:20.690719 10.1.12.11.8001 > 10.2.10.212.56690: P 6:9(3) ack 1 win 65529 <nop,nop,timestamp 27146230 3617562713> (DF)
19:03:20.690727 10.2.10.212.56690 > 10.1.12.11.8001: . ack 9 win 181 <nop,nop,timestamp 3617563744 27146230> (DF)

10.1.12.11 is the Win2k box, 10.2.10.212 is the Linux box. The socket
buffer sizes are the defaults, so the scaling is most likely 2^2. The
packets being exchanged at this point are just heartbeats.

On Tuesday I can try to capture a full session from the very begining, if
you think it would help.

Thanks,
-Ion

2005-09-28 16:31:55

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

Hi Alexey,

On Fri, 2 Sep 2005, Alexey Kuznetsov wrote:

> Anyway, ignoring this puzzle, the following patch for 2.4 should help.
>
>
> --- net/ipv4/tcp_input.c.orig 2003-02-20 20:38:39.000000000 +0300
> +++ net/ipv4/tcp_input.c 2005-09-02 22:28:00.845952888 +0400
> @@ -343,8 +343,6 @@
> app_win -= tp->ack.rcv_mss;
> app_win = max(app_win, 2U*tp->advmss);
>
> - if (!ofo_win)
> - tp->window_clamp = min(tp->window_clamp, app_win);
> tp->rcv_ssthresh = min(tp->window_clamp, 2U*tp->advmss);
> }
> }

I'm very happy to report that the above patch, applied to 2.6.12.6, seems
to have cured the TCP window problem we were experiencing. I've been
testing it extensively over the last 4 weeks, and I have yet to see any
repeats of the previous behavior.

The TCP window now usually settles to around 4k (unscaled, 16k scaled),
which is smallish but good enough for our purposes.

Thanks,
-Ion

2005-09-29 15:17:46

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

Hello!

> >Anyway, ignoring this puzzle, the following patch for 2.4 should help.
> >
> >
> >--- net/ipv4/tcp_input.c.orig 2003-02-20 20:38:39.000000000 +0300
> >+++ net/ipv4/tcp_input.c 2005-09-02 22:28:00.845952888 +0400
> >@@ -343,8 +343,6 @@
> > app_win -= tp->ack.rcv_mss;
> > app_win = max(app_win, 2U*tp->advmss);
> >
> >- if (!ofo_win)
> >- tp->window_clamp = min(tp->window_clamp, app_win);
> > tp->rcv_ssthresh = min(tp->window_clamp, 2U*tp->advmss);
> > }
> >}
>
> I'm very happy to report that the above patch, applied to 2.6.12.6, seems
> to have cured the TCP window problem we were experiencing.

Good. I think the patch is to be applied to all mainstream kernels.

The only obstacle is the second report by Guillaume Autran <[email protected]>,
which has some allied characteristics, but after analysis it is something
impossible, read, cryptic and severe bug. :-( I did not get a responce
to the last query, so the investigation stalled.

Alexey

2005-09-29 15:33:41

by Guillaume Autran

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

Hi!

Sorry Alexey for keeping it quiet but I got pulled away to some other duties for
the past 3 weeks.

Anyway, the similar problem I was reporting has not been seen since that last
incident a month ago. We did change, on our application side, some of the
parameters (aka SO_RCVBUF) that did not need to be set in the first place (bug
on our side).

This plus your patch seem to have resolve the issues we were having. So, it's
all good !

Thanks again..
Guillaume.

Alexey Kuznetsov wrote:
> Hello!
>
>
>>>Anyway, ignoring this puzzle, the following patch for 2.4 should help.
>>>
>>>
>>>--- net/ipv4/tcp_input.c.orig 2003-02-20 20:38:39.000000000 +0300
>>>+++ net/ipv4/tcp_input.c 2005-09-02 22:28:00.845952888 +0400
>>>@@ -343,8 +343,6 @@
>>> app_win -= tp->ack.rcv_mss;
>>> app_win = max(app_win, 2U*tp->advmss);
>>>
>>>- if (!ofo_win)
>>>- tp->window_clamp = min(tp->window_clamp, app_win);
>>> tp->rcv_ssthresh = min(tp->window_clamp, 2U*tp->advmss);
>>> }
>>>}
>>
>>I'm very happy to report that the above patch, applied to 2.6.12.6, seems
>>to have cured the TCP window problem we were experiencing.
>
>
> Good. I think the patch is to be applied to all mainstream kernels.
>
> The only obstacle is the second report by Guillaume Autran <[email protected]>,
> which has some allied characteristics, but after analysis it is something
> impossible, read, cryptic and severe bug. :-( I did not get a responce
> to the last query, so the investigation stalled.
>
> Alexey
>

--
=======================================
Guillaume Autran
Senior Software Engineer
MRV Communications, Inc.
Tel: (978) 952-4932 office
=======================================

2005-09-29 16:05:41

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

On Thursday 29 September 2005 11:17 am, Alexey Kuznetsov wrote:
> Hello!
>
> > >Anyway, ignoring this puzzle, the following patch for 2.4 should help.
> > >
> > >
> > >--- net/ipv4/tcp_input.c.orig 2003-02-20 20:38:39.000000000 +0300
> > >+++ net/ipv4/tcp_input.c 2005-09-02 22:28:00.845952888 +0400
> > >@@ -343,8 +343,6 @@
> > > app_win -= tp->ack.rcv_mss;
> > > app_win = max(app_win, 2U*tp->advmss);
> > >
> > >- if (!ofo_win)
> > >- tp->window_clamp = min(tp->window_clamp, app_win);
> > > tp->rcv_ssthresh = min(tp->window_clamp, 2U*tp->advmss);
> > > }
> > >}
> >
> > I'm very happy to report that the above patch, applied to 2.6.12.6, seems
> > to have cured the TCP window problem we were experiencing.
>
> Good. I think the patch is to be applied to all mainstream kernels.

Has anyone looked at the patch I sent out on Sept 9? It goes a few steps
further, addressing some additional problems. Original message below.

Thanks,
-John

-----

This is a patch for discussion addressing some receive buffer growing issues. ?
This is partially related to the thread "Possible BUG in IPv4 TCP window
handling..." last week.

Specifically it addresses the problem of an interaction between rcvbuf
moderation (receiver autotuning) and rcv_ssthresh. ?The problem occurs when
sending small packets to a receiver with a larger MTU. ?(A very common case I
have is a host with a 1500 byte MTU sending to a host with a 9k MTU.) ?In
such a case, the rcv_ssthresh code is targeting a window size corresponding
to filling up the current rcvbuf, not taking into account that the new rcvbuf
moderation may increase the rcvbuf size.

One hunk makes rcv_ssthresh use tcp_rmem[2] as the size target rather than
rcvbuf. ?The other changes the behavior when it overflows its memory bounds
with in-order data so that it tries to grow rcvbuf (the same as with
out-of-order data).

These changes should help my problem of mixed MTUs, and should also help the
case from last week's thread I think. ?(In both cases though you still need
tcp_rmem[2] to be set much larger than the TCP window.) ?One question is if
this is too aggressive at trying to increase rcvbuf if it's under memory
stress.

? -John

Signed-off-by: John Heffner <[email protected]>

Attachments:

(No filename) (2.21 kB)
rcv_ssthresh.diff (1.96 kB)
Download all attachments

2005-09-29 18:17:55

[permalink] [raw]

Subject: Re: Possible BUG in IPv4 TCP window handling, all recent 2.4.x/2.6.x kernels

From: John Heffner <[email protected]>
Date: Thu, 29 Sep 2005 12:04:28 -0400

> Has anyone looked at the patch I sent out on Sept 9? It goes a few steps
> further, addressing some additional problems. Original message below.

It's in my inbox pending review, so it's not forgotten :-)

My gut instinct right now is that we should put Alexey's
2-liner in for 2.6.14 et al. then be thinking about your
scheme for 2.6.15

2005-09-30 00:29:41