Major changes: Added thin-stream info in new file:
Documentation/networking/tcp-thin.txt
Signed-off-by: Andreas Petlund <[email protected]>
---
Documentation/networking/tcp-thin.txt | 47 +++++++++++++++++++++++++++++++++
include/net/tcp.h | 7 +++++
2 files changed, 54 insertions(+), 0 deletions(-)
create mode 100644 Documentation/networking/tcp-thin.txt
diff --git a/Documentation/networking/tcp-thin.txt b/Documentation/networking/tcp-thin.txt
new file mode 100644
index 0000000..151e229
--- /dev/null
+++ b/Documentation/networking/tcp-thin.txt
@@ -0,0 +1,47 @@
+Thin-streams and TCP
+====================
+A wide range of Internet-based services that use reliable transport
+protocols display what we call thin-stream properties. This means
+that the application sends data with such a low rate that the
+retransmission mechanisms of the transport protocol are not fully
+effective. In time-dependent scenarios (like online games, control
+systems, stock trading etc.) where the user experience depends
+on the data delivery latency, packet loss can be devastating for
+the service quality. Extreme latencies are caused by TCP's
+dependency on the arrival of new data from the application to trigger
+retransmissions effectively through fast retransmit instead of
+waiting for long timeouts.
+
+After analysing a large number of time-dependent interactive
+applications, we have seen that they often produce thin streams
+and also stay with this traffic pattern throughout its entire
+lifespan. The combination of time-dependency and the fact that the
+streams provoke high latencies when using TCP is unfortunate.
+
+In order to reduce application-layer latency when packets are lost,
+a set of mechanisms has been made, which address these latency issues
+for thin streams. In short, if the kernel detects a thin stream,
+the retransmission mechanisms are modified in the following manner:
+
+1) If the stream is thin, fast retransmit on the first dupACK.
+2) If the stream is thin, do not apply exponential backoff.
+
+These enhancements are applied only if the stream is detected as
+thin. This is accomplished by defining a threshold for the number
+of packets in flight. If there are less than 4 packets in flight,
+fast retransmissions can not be triggered, and the stream is prone
+to experience high retransmission latencies.
+
+Since these mechanisms are targeted at time-dependent applications,
+they must be specifically activated by the application using the
+TCP_THIN_LINEAR_TIMEOUTS and TCP_THIN_DUPACK IOCTLS or the
+tcp_thin_linear_timeouts and tcp_thin_dupack sysctls. Both
+modifications are turned off by default.
+
+References
+==========
+More information on the modifications, as well as a wide range of
+experimental data can be found here:
+"Improving latency for interactive, thin-stream applications over
+reliable transport"
+http://simula.no/research/nd/publications/Simula.nd.477/simula_pdf_file
diff --git a/include/net/tcp.h b/include/net/tcp.h
index 87d164b..e5e2056 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -1386,6 +1386,13 @@ static inline void tcp_highest_sack_combine(struct sock *sk,
tcp_sk(sk)->highest_sack = new;
}
+/* Determines whether this is a thin stream (which may suffer from
+ * increased latency). Used to trigger latency-reducing mechanisms.*/
+static inline unsigned int tcp_stream_is_thin(struct tcp_sock *tp)
+{
+ return tp->packets_out < 4 && !tcp_in_initial_slowstart(tp);
+}
+
/* /proc */
enum tcp_seq_states {
TCP_SEQ_STATE_LISTENING,
--
1.6.3.3
From: Andreas Petlund <[email protected]>
Date: Tue, 16 Feb 2010 15:40:32 +0100
>
> +/* Determines whether this is a thin stream (which may suffer from
> + * increased latency). Used to trigger latency-reducing mechanisms.*/
> +static inline unsigned int tcp_stream_is_thin(struct tcp_sock *tp)
> +{
> + return tp->packets_out < 4 && !tcp_in_initial_slowstart(tp);
> +}
> +
> /* /proc */
Please format comments:
/* Like this. */
or:
/* Like this.
* And this.
*/
Thanks.
Hi!
> +After analysing a large number of time-dependent interactive
> +applications, we have seen that they often produce thin streams
> +and also stay with this traffic pattern throughout its entire
> +lifespan. The combination of time-dependency and the fact that the
> +streams provoke high latencies when using TCP is unfortunate.
> +
> +In order to reduce application-layer latency when packets are lost,
> +a set of mechanisms has been made, which address these latency issues
> +for thin streams. In short, if the kernel detects a thin stream,
> +the retransmission mechanisms are modified in the following manner:
> +
> +1) If the stream is thin, fast retransmit on the first dupACK.
> +2) If the stream is thin, do not apply exponential backoff.
2) seems very dangerous/unfair. If network congestion is caused just
by thin streams, will the network just fall apart?
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Am 21.02.2010 um 11:21 schrieb Pavel Machek:
> Hi!
>
>> +After analysing a large number of time-dependent interactive
>> +applications, we have seen that they often produce thin streams
>> +and also stay with this traffic pattern throughout its entire
>> +lifespan. The combination of time-dependency and the fact that the
>> +streams provoke high latencies when using TCP is unfortunate.
>> +
>> +In order to reduce application-layer latency when packets are lost,
>> +a set of mechanisms has been made, which address these latency issues
>> +for thin streams. In short, if the kernel detects a thin stream,
>> +the retransmission mechanisms are modified in the following manner:
>> +
>> +1) If the stream is thin, fast retransmit on the first dupACK.
>> +2) If the stream is thin, do not apply exponential backoff.
>
> 2) seems very dangerous/unfair. If network congestion is caused just
> by thin streams, will the network just fall apart?
and 1) can also be dangerous if we have reordering on the path.
I strongly suggest that we discuss Andreas' idea on IETF TCPM *before*
we integrate it in the kernel and enable it for everyone
Alex,
as an netdev reader and TCPM member
>
>
> --
> (english) http://www.livejournal.com/~pavelmachek
> (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
//
// Dipl.-Inform. Alexander Zimmermann
// Department of Computer Science, Informatik 4
// RWTH Aachen University
// Ahornstr. 55, 52056 Aachen, Germany
// phone: (49-241) 80-21422, fax: (49-241) 80-22220
// email: [email protected]
// web: http://www.umic-mesh.net
//
On 2010-2-21, at 13:23, Alexander Zimmermann wrote:
> I strongly suggest that we discuss Andreas' idea on IETF TCPM *before*
> we integrate it in the kernel and enable it for everyone
Agree with Alexander. It is not at all clear to me that these changes are safe for non-exerimental use in the wider Internet.
Lars
* Pavel Machek | 2010-02-21 11:21:03 [+0100]:
>> +1) If the stream is thin, fast retransmit on the first dupACK.
>> +2) If the stream is thin, do not apply exponential backoff.
>
>2) seems very dangerous/unfair. If network congestion is caused just
>by thin streams, will the network just fall apart?
I question the fairness of this modification! The modification sounds _really_
far-reaching. Did you analyse the behaviour of this modification? What about
fairness analysis? Great work is available in the network simulator sector for
this kind of analysis (e.g. jain's fairness index).
HGN
--
Hagen Paul Pfeifer <[email protected]> || http://jauu.net/
Telephone: +49 174 5455209 || Key Id: 0x98350C22
Key Fingerprint: 490F 557B 6C48 6D7E 5706 2EA2 4A22 8D45 9835 0C22
On Sun, 21 Feb 2010, Pavel Machek wrote:
> Hi!
>
> > +After analysing a large number of time-dependent interactive
> > +applications, we have seen that they often produce thin streams
> > +and also stay with this traffic pattern throughout its entire
> > +lifespan. The combination of time-dependency and the fact that the
> > +streams provoke high latencies when using TCP is unfortunate.
> > +
> > +In order to reduce application-layer latency when packets are lost,
> > +a set of mechanisms has been made, which address these latency issues
> > +for thin streams. In short, if the kernel detects a thin stream,
> > +the retransmission mechanisms are modified in the following manner:
> > +
> > +1) If the stream is thin, fast retransmit on the first dupACK.
> > +2) If the stream is thin, do not apply exponential backoff.
>
> 2) seems very dangerous/unfair. If network congestion is caused just
> by thin streams, will the network just fall apart?
...Network will not fall apart. Two possible cases:
a) cwnd > 1 segment, on RTO the window is reduced, thus the sender backs
off regardless of exponential backoff.
b) All flows have window of 1 already... Well, what do you suggest? I'd
say that obviously the network is seriously underprovisioned for the
workload and since the bottleneck can only serve some of them anyway
dropping the rest, no useless work gets done at the bottleneck. RTT
estimator then gets a new samples whenever a flow belongs into the lucky
group. In case any unnecessary retransmission happened (if there was
very dramatic and sudden increase transient in the workload) they won't
happen thereafter (unless we increase the workload toward infinity).
...Thus no problem of "falling apart".
--
i.
On Sun, 21 Feb 2010, Alexander Zimmermann wrote:
>
> Am 21.02.2010 um 11:21 schrieb Pavel Machek:
>
> > Hi!
> >
> >> +After analysing a large number of time-dependent interactive
> >> +applications, we have seen that they often produce thin streams
> >> +and also stay with this traffic pattern throughout its entire
> >> +lifespan. The combination of time-dependency and the fact that the
> >> +streams provoke high latencies when using TCP is unfortunate.
> >> +
> >> +In order to reduce application-layer latency when packets are lost,
> >> +a set of mechanisms has been made, which address these latency issues
> >> +for thin streams. In short, if the kernel detects a thin stream,
> >> +the retransmission mechanisms are modified in the following manner:
> >> +
> >> +1) If the stream is thin, fast retransmit on the first dupACK.
> >> +2) If the stream is thin, do not apply exponential backoff.
> >
> > 2) seems very dangerous/unfair. If network congestion is caused just
> > by thin streams, will the network just fall apart?
>
> and 1) can also be dangerous if we have reordering on the path.
>
> I strongly suggest that we discuss Andreas' idea on IETF TCPM *before*
> we integrate it in the kernel and enable it for everyone
What difference you see with 1) and early rexmit when cwnd = 2, the latter
being afaict "discussed already" on TCPM?
--
i.
From: Lars Eggert <[email protected]>
Date: Sun, 21 Feb 2010 22:04:46 +0200
> Agree with Alexander. It is not at all clear to me that these
> changes are safe for non-exerimental use in the wider Internet.
It's off by default and we provide all sorts of experimental
congestion control algorithms which can cause just as much
if not more trouble than these thin-stream bits.
There is no problem integrating these changes.
If it's off by default there is no issue.
On 2010-2-22, at 1:03, David Miller wrote:
> From: Lars Eggert <[email protected]>
> Date: Sun, 21 Feb 2010 22:04:46 +0200
>
>> Agree with Alexander. It is not at all clear to me that these
>> changes are safe for non-exerimental use in the wider Internet.
>
> It's off by default and we provide all sorts of experimental
> congestion control algorithms which can cause just as much
> if not more trouble than these thin-stream bits.
>
> There is no problem integrating these changes.
Lars Eggert <[email protected]> writes:
> If it's off by default there is no issue.
Still a sufficient large number of people might turn it on
and then the network would melt.
IMHO TCP changes with badly understood congestion behaviour
should not be merged until the necessary analysis/simulation etc.
is done.
-Andi
--
[email protected] -- Speaking for myself only.
Am 22.02.2010 um 16:17 schrieb Andi Kleen:
> Lars Eggert <[email protected]> writes:
>
>> If it's off by default there is no issue.
>
> Still a sufficient large number of people might turn it on
> and then the network would melt.
>
> IMHO TCP changes with badly understood congestion behaviour
> should not be merged until the necessary analysis/simulation etc.
> is done.
>
> -Andi
100% ack.
See for example here: http://www.umic-mesh.net/~zimmermann/linux.png
It's a screenshot of the ps3mediaserver. Since it's a streaming server, it's
very useful to deactivate Nagle :-)
Alex
>
> --
> [email protected] -- Speaking for myself only.
//
// Dipl.-Inform. Alexander Zimmermann
// Department of Computer Science, Informatik 4
// RWTH Aachen University
// Ahornstr. 55, 52056 Aachen, Germany
// phone: (49-241) 80-21422, fax: (49-241) 80-22220
// email: [email protected]
// web: http://www.umic-mesh.net
//
On 22. feb. 2010 16:17, Andi Kleen wrote:
> Lars Eggert <[email protected]> writes:
>
>> If it's off by default there is no issue.
>
> Still a sufficient large number of people might turn it on
> and then the network would melt.
>
> IMHO TCP changes with badly understood congestion behaviour
> should not be merged until the necessary analysis/simulation etc.
> is done.
>
> -Andi
Graphs from our experiments can be found here:
http://simula.no/research/nd/publications/Simula.nd.477/simula_pdf_file
Section 5.2.5 covers experiments on fairness-issues. We have found no
adverse effects of the modifications submitted in this patch set (mFR
and LT in the plot), but further testing cannot hurt.
Best regards,
Andreas
On 22. feb. 2010 16:17, Andi Kleen wrote:
> Lars Eggert <[email protected]> writes:
>
>> If it's off by default there is no issue.
>
> Still a sufficient large number of people might turn it on
> and then the network would melt.
>
> IMHO TCP changes with badly understood congestion behaviour
> should not be merged until the necessary analysis/simulation etc.
> is done.
>
> -Andi
>
I also encourage you to read the discussions from the first posted
version of the patch set, since I here reference a lot of the
supporting research and related material from other researchers.
http://thread.gmane.org/gmane.linux.network/141394
http://thread.gmane.org/gmane.linux.network/141395
http://thread.gmane.org/gmane.linux.network/141396
http://thread.gmane.org/gmane.linux.network/141397
Cheers,
Andreas
On Mon, 22 Feb 2010 16:17:33 +0100, Andi Kleen <[email protected]> wrote:
> Lars Eggert <[email protected]> writes:
>
>> If it's off by default there is no issue.
>
> Still a sufficient large number of people might turn it on
> and then the network would melt.
Then we should apply the same policy as used for the congestion control
algorithms!? Restrict the usage to change the standard behavior via Kconfig
(e.g. menuconfig TCP_CONG_ADVANCED) or even more restricted demand for
CAP_NET_ADMIN.
Hagen
From: Andi Kleen <[email protected]>
Date: Mon, 22 Feb 2010 16:17:33 +0100
> Lars Eggert <[email protected]> writes:
>
>> If it's off by default there is no issue.
>
> Still a sufficient large number of people might turn it on
> and then the network would melt.
And they can do the same by changing the initial congestion window
setting, which we also support.
The foundation of your argument is as solid as quicksand.
