2001-11-06 11:20:14

by Marc Lehmann

[permalink] [raw]
Subject: ip_conntrack & timing out of connections

linux-2.4.13-ac5 (other versions untested) has this peculiar behaviour: If I
"killall -STOP thttpd", I, of course, still get connection requests which
usually time out:

tcp 238 0 217.227.148.85:80 213.76.191.129:3120 CLOSE_WAIT
tcp 162 0 217.227.148.85:80 213.76.191.129:3128 CLOSE_WAIT
tcp 238 0 217.227.148.85:80 213.76.191.129:3136 CLOSE_WAIT
tcp 162 0 217.227.148.85:80 213.76.191.129:3152 CLOSE_WAIT
tcp 134 0 217.227.148.85:80 66.42.121.15:3305 CLOSE_WAIT
tcp 162 0 217.227.148.85:80 213.76.191.129:3160 CLOSE_WAIT
tcp 279 0 217.227.148.85:80 62.83.11.19:2742 CLOSE_WAIT

however, after some time, I get many of these messages:

Nov 6 02:39:55 doom kernel: ip_conntrack: table full, dropping packet.

/proc/net/ip_conntrack has lots of connections like these:

tcp 6 430665 ESTABLISHED src=213.76.191.129 dst=217.227.148.85 sport=3881 dport=80 src=217.227.148.85 dst=213.76.191.129 sport=80 dport=388 1 [ASSURED] use=1

that is, connections to port 80. a grep dport=80 in ip_conntrack gives me
3768 lines, where netstat -t only shows 159 connections, so it seems that
conntrack has a problems with time-outs (or something similar).

--
-----==- |
----==-- _ |
---==---(_)__ __ ____ __ Marc Lehmann +--
--==---/ / _ \/ // /\ \/ / [email protected] |e|
-=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+
The choice of a GNU generation |
|


2001-11-06 13:07:58

by Rasmus Bøg Hansen

[permalink] [raw]
Subject: Re: ip_conntrack & timing out of connections

On Tue, 6 Nov 2001 [email protected] wrote:

> Nov 6 02:39:55 doom kernel: ip_conntrack: table full, dropping packet.

You probably need to do something like:

# We need a lot of concurrent connections
echo 65536 > /proc/sys/net/ipv4/ip_conntrack_max

(or how many you will need). Be aware that it will use up more memory -
the netfilter docs can tell you how much.

Rasmus

--
-- [ Rasmus 'M?ffe' B?g Hansen ] ---------------------------------------
If you only have a hammer
everything looks like a nail
--------------------------------- [ moffe at amagerkollegiet dot dk ] --

2001-11-06 19:05:18

by David Lang

[permalink] [raw]
Subject: Re: ip_conntrack & timing out of connections

the tcp dimeout is 60 seconds and the ip_conntrack timeout is 120 seconds.

I ran into this myself a couple days ago and have sent the info to Rusty
and to the netfilter mailing list (although only to the list this morning
so I don't know if there is a response there yet)

David Lang

On Tue, 6 Nov 2001 [email protected] wrote:

> Date: Tue, 6 Nov 2001 12:19:47 +0100
> From: [email protected]
> To: [email protected]
> Subject: ip_conntrack & timing out of connections
>
> linux-2.4.13-ac5 (other versions untested) has this peculiar behaviour: If I
> "killall -STOP thttpd", I, of course, still get connection requests which
> usually time out:
>
> tcp 238 0 217.227.148.85:80 213.76.191.129:3120 CLOSE_WAIT
> tcp 162 0 217.227.148.85:80 213.76.191.129:3128 CLOSE_WAIT
> tcp 238 0 217.227.148.85:80 213.76.191.129:3136 CLOSE_WAIT
> tcp 162 0 217.227.148.85:80 213.76.191.129:3152 CLOSE_WAIT
> tcp 134 0 217.227.148.85:80 66.42.121.15:3305 CLOSE_WAIT
> tcp 162 0 217.227.148.85:80 213.76.191.129:3160 CLOSE_WAIT
> tcp 279 0 217.227.148.85:80 62.83.11.19:2742 CLOSE_WAIT
>
> however, after some time, I get many of these messages:
>
> Nov 6 02:39:55 doom kernel: ip_conntrack: table full, dropping packet.
>
> /proc/net/ip_conntrack has lots of connections like these:
>
> tcp 6 430665 ESTABLISHED src=213.76.191.129 dst=217.227.148.85 sport=3881 dport=80 src=217.227.148.85 dst=213.76.191.129 sport=80 dport=388 1 [ASSURED] use=1
>
> that is, connections to port 80. a grep dport=80 in ip_conntrack gives me
> 3768 lines, where netstat -t only shows 159 connections, so it seems that
> conntrack has a problems with time-outs (or something similar).
>
> --
> -----==- |
> ----==-- _ |
> ---==---(_)__ __ ____ __ Marc Lehmann +--
> --==---/ / _ \/ // /\ \/ / [email protected] |e|
> -=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+
> The choice of a GNU generation |
> |
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

2001-11-07 18:55:57

by Alexey Kuznetsov

[permalink] [raw]
Subject: Re: ip_conntrack & timing out of connections

Hello!

> the tcp dimeout is 60 seconds and the ip_conntrack timeout is 120 seconds.

This is absolutely different case.


> > From: [email protected]
...
> > linux-2.4.13-ac5 (other versions untested) has this peculiar behaviour: If I
> > "killall -STOP thttpd", I, of course, still get connection requests which
> > usually time out:
> >
> > tcp 238 0 217.227.148.85:80 213.76.191.129:3120 CLOSE_WAIT

Blatant lie. Such connections cannot timeout. If they do, kernel really
have fatal bug.


> > Nov 6 02:39:55 doom kernel: ip_conntrack: table full, dropping packet.
> >
> > /proc/net/ip_conntrack has lots of connections like these:
> >
> > tcp 6 430665 ESTABLISHED src=213.76.191.129 dst=217.227.148.85 sport=3881 dport=80 src=217.227.148.85 dst=213.76.191.129 sport=80 dport=388 1 [ASSURED] use=1

It is absolutely right. Established connections must not timeout.

Alexey

2001-11-07 19:42:37

by Trever L. Adams

[permalink] [raw]
Subject: Re: ip_conntrack & timing out of connections

On Wed, 2001-11-07 at 13:55, [email protected] wrote:
> > > From: [email protected]
> ...
> > > linux-2.4.13-ac5 (other versions untested) has this peculiar behaviour: If I
> > > "killall -STOP thttpd", I, of course, still get connection requests which
> > > usually time out:
> > >
> > > tcp 238 0 217.227.148.85:80 213.76.191.129:3120 CLOSE_WAIT
>
> Blatant lie. Such connections cannot timeout. If they do, kernel really
> have fatal bug.
>

Then the kernel (iptables or what not) has a huge fatal bug. I have
seen this happen as well. The firewall then catches all of these 'ACK
FIN' etc. This is getting more rare for me and usually takes a moderate
to heavy load (for link capacity) before it starts happening, but it
does happen.

Trever


2001-11-12 07:56:34

by Harald Welte

[permalink] [raw]
Subject: Re: ip_conntrack & timing out of connections

On Tue, Nov 06, 2001 at 12:19:47PM +0100, Marc A. Lehmann wrote:

> however, after some time, I get many of these messages:
>
> Nov 6 02:39:55 doom kernel: ip_conntrack: table full, dropping packet.
>
> /proc/net/ip_conntrack has lots of connections like these:
>
> tcp 6 430665 ESTABLISHED src=213.76.191.129 dst=217.227.148.85 sport=3881 dport=80 src=217.227.148.85 dst=213.76.191.129 sport=80 dport=388 1 [ASSURED] use=1
>
> that is, connections to port 80. a grep dport=80 in ip_conntrack gives me
> 3768 lines, where netstat -t only shows 159 connections, so it seems that
> conntrack has a problems with time-outs (or something similar).

connection tracking keeps all TCP conntrack entries for 120 seconds after
completion of FIN <-> FIN closedown. This is the TIME_WAIT state of the
tcp protocol.

Maybe the linux tcp stack doesn't wait for 120 seconds, or some other
condition in the tcp stack makes the sockets disappear from the netstat -t
list.

--
Live long and prosper
- Harald Welte / [email protected] http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M-
V-- PS+ PE-- Y+ PGP++ t++ 5-- !X !R tv-- b+++ DI? !D G+ e* h+ r% y+(*)