I was DDoS'd today while away and came home to find the firewall unable to
do anything network related (although my connection to irc was still
working oddly). a quick dmesg showed the problem.
ip_conntrack: maximum limit of 2048 entries exceeded
NET: 1 messages suppressed.
ip_conntrack: maximum limit of 2048 entries exceeded
NET: 3 messages suppressed.
ip_conntrack: maximum limit of 2048 entries exceeded
NAT: 0 dropping untracked packet c1e69980 6 192.168.1.2 -> 206.251.7.30
ip_conntrack: maximum limit of 2048 entries exceeded
NAT: 0 dropping untracked packet c1e69b60 6 192.168.1.2 -> 206.251.7.30
ip_conntrack: maximum limit of 2048 entries exceeded
That is a very small snippet of dmesg. It seems that ip_conntrack did not
flush or reset after the attack, even though everything was fine when i got
home. Keep in mind, this was a somewhat massive attack on my network here
but is only connected via a DSL line, it seems the attackers sent hundreds
or thousands of very small packets resulting in 21000 connection attempts
in a short amount of time. Is this a bug with ip_conntrack? this is
kernel version 2.4.0-test5, it's been up for 74 days. I had to reload
ip_conntrack to flush it and everything worked fine after that. Thanks
for any info.
ps. If this is a previously discovered bug, is it fixed in the latest
kernels?
On Wed, Nov 15, 2000 at 03:46:03PM -0500, safemode wrote:
> I was DDoS'd today while away and came home to find the firewall unable to
> do anything network related (although my connection to irc was still
> working oddly). a quick dmesg showed the problem.
> ip_conntrack: maximum limit of 2048 entries exceeded
[...]
I have also seen this happen on a box which ran test9. Apparently because of
it's long uptime, because the logs should no signs of an attack.
I guess conntrack forgets to flush some entries? Or maybe there is no way it can
recover from a full conntrack table? Is it maybe necessary to make the maximum
size a configurable option? Or a userspace conntrack daemon like the arpd?
I also see a lot of messages like this (on all 2.4 test kernels):
NAT: 0 dropping untracked packet c00643f0 1 131.211.122.89 -> 224.0.0.2
NAT: 0 dropping untracked packet c05468e0 1 131.211.122.89 -> 224.0.0.2
NAT: 0 dropping untracked packet c0064760 1 131.211.122.31 -> 224.0.0.2
Turning of multicast on the respective network interface does not stop these
messages, but anyway they seem rather annoying to me :)
-------------------------------------------
Met vriendelijke groet / with kind regards,
Guus Sliepen <[email protected]>
-------------------------------------------
See also: http://tinc.nl.linux.org/
http://www.kernelbench.org/
-------------------------------------------
On Wed, 15 Nov 2000 16:19:23 Guus Sliepen wrote:
> On Wed, Nov 15, 2000 at 03:46:03PM -0500, safemode wrote:
>
> > I was DDoS'd today while away and came home to find the firewall unable
> to
> > do anything network related (although my connection to irc was still
> > working oddly). a quick dmesg showed the problem.
> > ip_conntrack: maximum limit of 2048 entries exceeded
> [...]
>
> I have also seen this happen on a box which ran test9. Apparently because
> of
> it's long uptime, because the logs should no signs of an attack.
>
> I guess conntrack forgets to flush some entries? Or maybe there is no way
> it can
> recover from a full conntrack table? Is it maybe necessary to make the
> maximum
> size a configurable option? Or a userspace conntrack daemon like the
> arpd?
I think something is wrong if the ip_conntrack module does not
flush it's table after the connections and all that stop. I understand why
it does this during the attack...but it doesn't make sense why these tables
are kept long after. A userspace tool is not something i think is needed,
this piece of code should be in the module, it is either not correctly
coded or missing entirely.
> I also see a lot of messages like this (on all 2.4 test kernels):
>
> NAT: 0 dropping untracked packet c00643f0 1 131.211.122.89 -> 224.0.0.2
> NAT: 0 dropping untracked packet c05468e0 1 131.211.122.89 -> 224.0.0.2
> NAT: 0 dropping untracked packet c0064760 1 131.211.122.31 -> 224.0.0.2
>
> Turning of multicast on the respective network interface does not stop
> these
> messages, but anyway they seem rather annoying to me :)
Everyone has seen that :) ... that's not exactly what i was talking about
the main error message i was worried about was the "ip_conntrack: maximum
limit of 2048 entries exceeded" when there was absolutely not traffic
coming in and the attack was long since over. I think this is a fairly
major bug with the module since it made the firewall inoperable until i
reloaded the module.. this DDoS could be repeated on any linux box that is
not babysat 24/7 it seems. My firewall drops everything so all the
attacker needs to do is get a bunch of sources to send packets (specific?
not sure) rapidly enough to fill the ip_conntrack table and your site
becomes offline. any other ideas?
On Wed, Nov 15, 2000 at 04:34:50PM -0500, safemode wrote:
> On Wed, 15 Nov 2000 16:19:23 Guus Sliepen wrote:
> > On Wed, Nov 15, 2000 at 03:46:03PM -0500, safemode wrote:
> >
> > > I was DDoS'd today while away and came home to find the firewall unable
> > to
> > > do anything network related (although my connection to irc was still
> > > working oddly). a quick dmesg showed the problem.
> > > ip_conntrack: maximum limit of 2048 entries exceeded
> > [...]
> >
> > I have also seen this happen on a box which ran test9. Apparently because
> > of
> > it's long uptime, because the logs should no signs of an attack.
safemode and I discussed this and we tried to find an answer in the kernel
source. However, the chain of called functions is too long to determine where
exactly the problem is. But most likely, because init_conntrack() can fail
(because it cannot free an entry, which is either because netfilter does not
dare to throw out entries with large timeouts (tcp connections have ridiculous
long timeouts btw, almost 2.3 days?!) or because IPS_CONFIRMED is not set), and
this failure is propagating back all the way to the tcp code, so that no new
sockets can be opened.
From our point of view, the conntrack stuff should be totally transparent to the
tcp/ip stack. Since this allows for a DoS attack, might be wise to fix this
before 2.4 comes out...
-------------------------------------------
Met vriendelijke groet / with kind regards,
Guus Sliepen <[email protected]>
-------------------------------------------
See also: http://tinc.nl.linux.org/
http://www.kernelbench.org/
-------------------------------------------
On Wed, 15 Nov 2000, Guus Sliepen wrote:
> > I was DDoS'd today while away and came home to find the firewall unable to
> > do anything network related (although my connection to irc was still
> > working oddly). a quick dmesg showed the problem.
> > ip_conntrack: maximum limit of 2048 entries exceeded
> [...]
>
> I have also seen this happen on a box which ran test9. Apparently because of
> it's long uptime, because the logs should no signs of an attack.
>
> I guess conntrack forgets to flush some entries? Or maybe there is no way it can
> recover from a full conntrack table? Is it maybe necessary to make the maximum
> size a configurable option? Or a userspace conntrack daemon like the arpd?
>From reading the sources I got the impression that the use count of
the ip_conntrack struct isn't decremented properly. This causes
destroy_conntrack() not to free ip_conntrack's - which results allocation
until the maximum (ip_conntrack_max), and failing to allocate new ones.
p.s. Get a popcorn when you're reading netfilter's sources - bumping into
a label like 'i_see_dead_people' is quite amusing...
--
Dan Aloni
[email protected]
On Thu, 16 Nov 2000, Dan Aloni wrote:
> On Wed, 15 Nov 2000, Guus Sliepen wrote:
>
> > > I was DDoS'd today while away and came home to find the firewall unable to
> > > do anything network related (although my connection to irc was still
> > > working oddly). a quick dmesg showed the problem.
> > > ip_conntrack: maximum limit of 2048 entries exceeded
> > [...]
> >
> > I have also seen this happen on a box which ran test9. Apparently because of
> > it's long uptime, because the logs should no signs of an attack.
> >
> > I guess conntrack forgets to flush some entries? Or maybe there is no way it can
> > recover from a full conntrack table? Is it maybe necessary to make the maximum
> > size a configurable option? Or a userspace conntrack daemon like the arpd?
>
> >From reading the sources I got the impression that the use count of
> the ip_conntrack struct isn't decremented properly. This causes
> destroy_conntrack() not to free ip_conntrack's - which results allocation
> until the maximum (ip_conntrack_max), and failing to allocate new ones.
I think I got something, icmp_error_track() increases the use count
(calling ip_conntrack_find_get()) when it returns with no error (not NULL).
Whoever calls icmp_error_track() and gets a valid pointer to ip_conntrack,
must call ip_conntrack_put() - look at ip_conntrack_in(), line 685, the
pointer is just used in a boolean expression without calling
ip_conntrack_put(). I'm not sure if other places needed fixing, but anyway
try this patch:
--- linux-2.4.0-test11-pre5/net/ipv4/netfilter/ip_conntrack_core.c Tue Nov 14 09:56:16 2000
+++ linux/net/ipv4/netfilter/ip_conntrack_core.c Thu Nov 16 01:35:13 2000
@@ -682,8 +682,10 @@
/* It may be an icmp error... */
if ((*pskb)->nh.iph->protocol == IPPROTO_ICMP
- && icmp_error_track(*pskb, &ctinfo, hooknum))
+ && (ct = icmp_error_track(*pskb, &ctinfo, hooknum))) {
+ ip_conntrack_put(ct);
return NF_ACCEPT;
+ }
if (!(ct = resolve_normal_ct(*pskb, proto,&set_reply,hooknum,&ctinfo)))
/* Not valid part of a connection */
--
Dan Aloni
[email protected]
vegae:/usr/src/linux# grep -r ./* --regexp="IPS_CON" | grep "define"
./include/linux/elf.h:#define DT_MIPS_CONFLICT 0x70000008
./include/linux/elf.h:#define DT_MIPS_CONFLICTNO 0x7000000b
./include/linux/elf.h:#define SHT_MIPS_CONFLICT 0x70000002
vegae:/usr/src/linux#
hmmm... looks like theriz no IPS_CONFIRMED
definition in test9...
> > > I have also seen this happen on a box which ran test9. Apparently because of
> > > it's long uptime, because the logs should no signs of an attack.
> > >
> > > I guess conntrack forgets to flush some entries? Or maybe there is no way it can
> > > recover from a full conntrack table? Is it maybe necessary to make the maximum
> > > size a configurable option? Or a userspace conntrack daemon like the arpd?
> >
> > From reading the sources I got the impression that the use count of
> > the ip_conntrack struct isn't decremented properly. This causes
> > destroy_conntrack() not to free ip_conntrack's - which results allocation
> > until the maximum (ip_conntrack_max), and failing to allocate new ones.
>
> I think I got something, icmp_error_track() increases the use count
> (calling ip_conntrack_find_get()) when it returns with no error (not NULL).
> Whoever calls icmp_error_track() and gets a valid pointer to ip_conntrack,
> must call ip_conntrack_put() - look at ip_conntrack_in(), line 685, the
> pointer is just used in a boolean expression without calling
> ip_conntrack_put(). I'm not sure if other places needed fixing, but anyway
> try this patch:
I'm not sure this works, since the use count also counts for skb's,
icmp_error_track(), makes the skb refer to this conntrack in case of
success, intentually not calling ip_conntrack_put().
So now I'm clueless, although I'm almost certain it's a use count
problem. I'd be happy to hear from Rusty or someone on the netfilter
mailing list about this.
--
Dan Aloni
[email protected]
In message <20001115154603.D4089@psuedomode> you write:
> I was DDoS'd today while away and came home to find the firewall unable to
> do anything network related (although my connection to irc was still
> working oddly). a quick dmesg showed the problem.
> ip_conntrack: maximum limit of 2048 entries exceeded
> NET: 1 messages suppressed.
> ip_conntrack: maximum limit of 2048 entries exceeded
> NET: 3 messages suppressed.
> ip_conntrack: maximum limit of 2048 entries exceeded
> NAT: 0 dropping untracked packet c1e69980 6 192.168.1.2 -> 206.251.7.30
> ip_conntrack: maximum limit of 2048 entries exceeded
> NAT: 0 dropping untracked packet c1e69b60 6 192.168.1.2 -> 206.251.7.30
> ip_conntrack: maximum limit of 2048 entries exceeded
Yes, I added connection assurance (which provides much more
intelligence about which connections should be dropped,
ie. established TCP connections get assured) in test6, for exactly
this reason.
Hope that helps,
Rusty.
--
Hacking time.
In message <[email protected]> you write
:
> I think I got something, icmp_error_track() increases the use count
> (calling ip_conntrack_find_get()) when it returns with no error (not NULL).
The reference count is now held by the skb.
Hope that helps,
Rusty.
--
Hacking time.