From: Menglong Dong <[email protected]>
In tcp_retransmit_timer(), a window shrunk connection will be regarded
as timeout if 'tcp_jiffies32 - tp->rcv_tstamp > TCP_RTO_MAX'. This is not
right all the time.
The retransmits will become zero-window probes in tcp_retransmit_timer()
if the 'snd_wnd==0'. Therefore, the icsk->icsk_rto will come up to
TCP_RTO_MAX sooner or later.
However, the timer is not precise enough, as it base on timer wheel.
Sorry that I am not good at timer, but I know the concept of time-wheel.
The longer of the timer, the rougher it will be. So the timeout is not
triggered after TCP_RTO_MAX, but 122877ms as I tested.
Therefore, 'tcp_jiffies32 - tp->rcv_tstamp > TCP_RTO_MAX' is always true
once the RTO come up to TCP_RTO_MAX, and the socket will die.
Fix this by replacing the 'tcp_jiffies32' with '(u32)icsk->icsk_timeout',
which is exact the timestamp of the timeout. Meanwhile, using
"max(tp->retrans_stamp, tp->rcv_tstamp)" as the last updated timestamp in
the receiving path, as "tp->rcv_tstamp" can restart from idle, then
tp->rcv_tstamp could already be a long time (minutes or hours) in the
past even on the first RTO.
Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
Link: https://lore.kernel.org/netdev/CADxym3YyMiO+zMD4zj03YPM3FBi-1LHi6gSD2XT8pyAMM096pg@mail.gmail.com/
Signed-off-by: Menglong Dong <[email protected]>
---
v2:
- consider the case of the connection restart from idle, as Neal comment
---
net/ipv4/tcp_timer.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c
index d45c96c7f5a4..e4b2d8706cae 100644
--- a/net/ipv4/tcp_timer.c
+++ b/net/ipv4/tcp_timer.c
@@ -454,6 +454,14 @@ static void tcp_fastopen_synack_timer(struct sock *sk, struct request_sock *req)
req->timeout << req->num_timeout, TCP_RTO_MAX);
}
+static bool tcp_rtx_probe0_timed_out(struct sock *sk)
+{
+ struct tcp_sock *tp = tcp_sk(sk);
+ u32 last_ts;
+
+ last_ts = max(tp->retrans_stamp, tp->rcv_tstamp);
+ return inet_csk(sk)->icsk_timeout - last_ts > TCP_RTO_MAX;
+}
/**
* tcp_retransmit_timer() - The TCP retransmit timeout handler
@@ -519,7 +527,7 @@ void tcp_retransmit_timer(struct sock *sk)
tp->snd_una, tp->snd_nxt);
}
#endif
- if (tcp_jiffies32 - tp->rcv_tstamp > TCP_RTO_MAX) {
+ if (tcp_rtx_probe0_timed_out(sk)) {
tcp_write_err(sk);
goto out;
}
--
2.40.1
On Mon, Aug 7, 2023 at 3:47 PM <[email protected]> wrote:
>
> From: Menglong Dong <[email protected]>
>
> In tcp_retransmit_timer(), a window shrunk connection will be regarded
> as timeout if 'tcp_jiffies32 - tp->rcv_tstamp > TCP_RTO_MAX'. This is not
> right all the time.
>
> The retransmits will become zero-window probes in tcp_retransmit_timer()
> if the 'snd_wnd==0'. Therefore, the icsk->icsk_rto will come up to
> TCP_RTO_MAX sooner or later.
>
> However, the timer is not precise enough, as it base on timer wheel.
> Sorry that I am not good at timer, but I know the concept of time-wheel.
Can you remove this line, can we keep focused on the actual patch instead ?
Regardless of timer-wheel, a timer can be delayed under load.
> The longer of the timer, the rougher it will be. So the timeout is not
> triggered after TCP_RTO_MAX, but 122877ms as I tested.
>
> Therefore, 'tcp_jiffies32 - tp->rcv_tstamp > TCP_RTO_MAX' is always true
> once the RTO come up to TCP_RTO_MAX, and the socket will die.
>
> Fix this by replacing the 'tcp_jiffies32' with '(u32)icsk->icsk_timeout',
> which is exact the timestamp of the timeout. Meanwhile, using
> "max(tp->retrans_stamp, tp->rcv_tstamp)" as the last updated timestamp in
> the receiving path, as "tp->rcv_tstamp" can restart from idle, then
> tp->rcv_tstamp could already be a long time (minutes or hours) in the
> past even on the first RTO.
>
> Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> Link: https://lore.kernel.org/netdev/CADxym3YyMiO+zMD4zj03YPM3FBi-1LHi6gSD2XT8pyAMM096pg@mail.gmail.com/
> Signed-off-by: Menglong Dong <[email protected]>
> ---
> v2:
> - consider the case of the connection restart from idle, as Neal comment
> ---
> net/ipv4/tcp_timer.c | 10 +++++++++-
> 1 file changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c
> index d45c96c7f5a4..e4b2d8706cae 100644
> --- a/net/ipv4/tcp_timer.c
> +++ b/net/ipv4/tcp_timer.c
> @@ -454,6 +454,14 @@ static void tcp_fastopen_synack_timer(struct sock *sk, struct request_sock *req)
> req->timeout << req->num_timeout, TCP_RTO_MAX);
> }
>
> +static bool tcp_rtx_probe0_timed_out(struct sock *sk)
> +{
> + struct tcp_sock *tp = tcp_sk(sk);
> + u32 last_ts;
> +
> + last_ts = max(tp->retrans_stamp, tp->rcv_tstamp);
u32 retrans_stamp; /* Timestamp of the last retransmit,
u32 rcv_tstamp; /* timestamp of last received ACK (for keepalives) */
Both fields receive tcp_jiffies32 values, which wrap every 2^32 ticks.
So max(A, B) won't work as you expect...
You need to use before(), after() or something like that.
https://en.wikipedia.org/wiki/Serial_number_arithmetic#General_Solution
> + return inet_csk(sk)->icsk_timeout - last_ts > TCP_RTO_MAX;
> +}
>
> /**
> * tcp_retransmit_timer() - The TCP retransmit timeout handler
> @@ -519,7 +527,7 @@ void tcp_retransmit_timer(struct sock *sk)
> tp->snd_una, tp->snd_nxt);
> }
> #endif
> - if (tcp_jiffies32 - tp->rcv_tstamp > TCP_RTO_MAX) {
> + if (tcp_rtx_probe0_timed_out(sk)) {
> tcp_write_err(sk);
> goto out;
> }
> --
> 2.40.1
>
On Mon, Aug 7, 2023 at 10:20 PM Eric Dumazet <[email protected]> wrote:
>
> On Mon, Aug 7, 2023 at 3:47 PM <[email protected]> wrote:
> >
> > From: Menglong Dong <[email protected]>
> >
> > In tcp_retransmit_timer(), a window shrunk connection will be regarded
> > as timeout if 'tcp_jiffies32 - tp->rcv_tstamp > TCP_RTO_MAX'. This is not
> > right all the time.
> >
> > The retransmits will become zero-window probes in tcp_retransmit_timer()
> > if the 'snd_wnd==0'. Therefore, the icsk->icsk_rto will come up to
> > TCP_RTO_MAX sooner or later.
> >
> > However, the timer is not precise enough, as it base on timer wheel.
>
> > Sorry that I am not good at timer, but I know the concept of time-wheel.
>
> Can you remove this line, can we keep focused on the actual patch instead ?
>
> Regardless of timer-wheel, a timer can be delayed under load.
>
Okay!
> > The longer of the timer, the rougher it will be. So the timeout is not
> > triggered after TCP_RTO_MAX, but 122877ms as I tested.
> >
> > Therefore, 'tcp_jiffies32 - tp->rcv_tstamp > TCP_RTO_MAX' is always true
> > once the RTO come up to TCP_RTO_MAX, and the socket will die.
> >
> > Fix this by replacing the 'tcp_jiffies32' with '(u32)icsk->icsk_timeout',
> > which is exact the timestamp of the timeout. Meanwhile, using
> > "max(tp->retrans_stamp, tp->rcv_tstamp)" as the last updated timestamp in
> > the receiving path, as "tp->rcv_tstamp" can restart from idle, then
> > tp->rcv_tstamp could already be a long time (minutes or hours) in the
> > past even on the first RTO.
> >
> > Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> > Link: https://lore.kernel.org/netdev/CADxym3YyMiO+zMD4zj03YPM3FBi-1LHi6gSD2XT8pyAMM096pg@mail.gmail.com/
> > Signed-off-by: Menglong Dong <[email protected]>
> > ---
> > v2:
> > - consider the case of the connection restart from idle, as Neal comment
> > ---
> > net/ipv4/tcp_timer.c | 10 +++++++++-
> > 1 file changed, 9 insertions(+), 1 deletion(-)
> >
> > diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c
> > index d45c96c7f5a4..e4b2d8706cae 100644
> > --- a/net/ipv4/tcp_timer.c
> > +++ b/net/ipv4/tcp_timer.c
> > @@ -454,6 +454,14 @@ static void tcp_fastopen_synack_timer(struct sock *sk, struct request_sock *req)
> > req->timeout << req->num_timeout, TCP_RTO_MAX);
> > }
> >
> > +static bool tcp_rtx_probe0_timed_out(struct sock *sk)
> > +{
> > + struct tcp_sock *tp = tcp_sk(sk);
> > + u32 last_ts;
> > +
> > + last_ts = max(tp->retrans_stamp, tp->rcv_tstamp);
>
> u32 retrans_stamp; /* Timestamp of the last retransmit,
> u32 rcv_tstamp; /* timestamp of last received ACK (for keepalives) */
>
> Both fields receive tcp_jiffies32 values, which wrap every 2^32 ticks.
>
> So max(A, B) won't work as you expect...
>
> You need to use before(), after() or something like that.
Now I know how the before()/after() works. I thought they
simply compared the arguments.
I'll use brefore/after here instead.
Thanks!
Menglong Dong
>
> https://en.wikipedia.org/wiki/Serial_number_arithmetic#General_Solution
>
>
> > + return inet_csk(sk)->icsk_timeout - last_ts > TCP_RTO_MAX;
> > +}
> >
> > /**
> > * tcp_retransmit_timer() - The TCP retransmit timeout handler
> > @@ -519,7 +527,7 @@ void tcp_retransmit_timer(struct sock *sk)
> > tp->snd_una, tp->snd_nxt);
> > }
> > #endif
> > - if (tcp_jiffies32 - tp->rcv_tstamp > TCP_RTO_MAX) {
> > + if (tcp_rtx_probe0_timed_out(sk)) {
> > tcp_write_err(sk);
> > goto out;
> > }
> > --
> > 2.40.1
> >