All,
I have a dual 3GHz Xeon machine with a 2.4.21 kernel and thousands (15k+) of
ipip tunnel interfaces. These are being used to tunnel traffic from remote
routers, over a private network, and handed off to a third party.
Creating the interfaces takes longer and longer the more there are. For the
first thousand or so interfaces, creation takes place at the rate of 40 per
second, later it drops to around 1 per second, then one every five seconds.
The tunnels are created thus:
ip tunnel add $interface mode ipip remote $peer local $eth0_address
ip addr add $eth0_address peer $lanip dev $interface
ip link set $interface arp off up
ip route add $remote_subnet dev $interface
where $interface is the name of the tunnel, $peer is the 'external'
interface on the remote router, $eth0_address is eth0's address on the
tunnel box, and $remote_subnet is the network we're tunneling.
Is it possible to speed up creation of the interfaces? Currently it takes
around 24 hours. Is there are more efficient way to handle a very large
number of IP-IP tunnels? Would upgrading to a 2.6 kernel be of use?
Is there a userspace program which would handle this application better than
using interfaces?
Peter.
From: Peter Hicks <[email protected]>
Date: Tue, 31 Oct 2006 09:25:50 +0000
[ Discussion belongs on [email protected], added to CC: ]
> I have a dual 3GHz Xeon machine with a 2.4.21 kernel and thousands (15k+) of
> ipip tunnel interfaces. These are being used to tunnel traffic from remote
> routers, over a private network, and handed off to a third party.
...
> Is it possible to speed up creation of the interfaces? Currently it takes
> around 24 hours. Is there are more efficient way to handle a very large
> number of IP-IP tunnels? Would upgrading to a 2.6 kernel be of use?
We just simply never imagined people would use IP tunnels on
this scale.
The following kernel patch is a quick hack that will get things to
work quickly for you, but longer term we need to add dynamic hash
table growth to this thing (and SIT tunnel, and IP GRE tunnel,
etc. etc. etc.)
diff --git a/net/ipv4/ipip.c b/net/ipv4/ipip.c
index 0c45565..78055cf 100644
--- a/net/ipv4/ipip.c
+++ b/net/ipv4/ipip.c
@@ -117,8 +117,8 @@ #include <net/ipip.h>
#include <net/inet_ecn.h>
#include <net/xfrm.h>
-#define HASH_SIZE 16
-#define HASH(addr) ((addr^(addr>>4))&0xF)
+#define HASH_SIZE 16384
+#define HASH(addr) ((addr^(addr>>14))&(HASH_SIZE - 1))
static int ipip_fb_tunnel_init(struct net_device *dev);
static int ipip_tunnel_init(struct net_device *dev);
David Miller wrote:
> From: Peter Hicks <[email protected]>
> Date: Tue, 31 Oct 2006 09:25:50 +0000
>
> [ Discussion belongs on [email protected], added to CC: ]
>
>
>> I have a dual 3GHz Xeon machine with a 2.4.21 kernel and thousands (15k+) of
>> ipip tunnel interfaces. These are being used to tunnel traffic from remote
>> routers, over a private network, and handed off to a third party.
>>
> ...
>
>> Is it possible to speed up creation of the interfaces? Currently it takes
>> around 24 hours. Is there are more efficient way to handle a very large
>> number of IP-IP tunnels? Would upgrading to a 2.6 kernel be of use?
>>
>
>
2.6 (and the associated 'ip' tool) does have some improvements for
showing very
large numbers of interfaces. I haven't tried more than a few thousand
though...
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
On Tue, 31 Oct 2006 01:31:54 -0800 (PST)
David Miller <[email protected]> wrote:
> From: Peter Hicks <[email protected]>
> Date: Tue, 31 Oct 2006 09:25:50 +0000
>
> [ Discussion belongs on [email protected], added to CC: ]
>
> > I have a dual 3GHz Xeon machine with a 2.4.21 kernel and thousands (15k+) of
> > ipip tunnel interfaces. These are being used to tunnel traffic from remote
> > routers, over a private network, and handed off to a third party.
> ...
> > Is it possible to speed up creation of the interfaces? Currently it takes
> > around 24 hours. Is there are more efficient way to handle a very large
> > number of IP-IP tunnels? Would upgrading to a 2.6 kernel be of use?
>
2.4 has a several N^2 searches for interfaces (and is in deep freeze by now).
2.6 had several changes to handle 1000's of interfaces.
--
Stephen Hemminger <[email protected]>
Peter Hicks wrote:
> All,
>
> I have a dual 3GHz Xeon machine with a 2.4.21 kernel and thousands (15k+) of
> ipip tunnel interfaces. These are being used to tunnel traffic from remote
> routers, over a private network, and handed off to a third party.
[]
> Is there a userspace program which would handle this application better than
> using interfaces?
Not that it may be suitable for your case because of various reasons (including
but not limited to your use of specific - IPIP - type of tunnels, interoperability
issues), but take a look at the tinc principles -- http://www.tinc-vpn.org/ . They
use single interface (based on tun driver) and a single select-loop-based userspace
program. Initially you configure routing to route ALL your peer's traffic to this
interface, and next tincd takes care of {dis,re}appearing peers, shortest pathes,
{un}reachability of certain networks and so on.
I don't know whenever their implementation scales up to 15K+ peers any better than
current in-kernel implementation, but I think it's easier to deal with this stuff
in userspace anyway. And the principles which are in the base of tinc are very..
interesting and are unique (as far as I know) to it, making this solution ideal for
certain setups.
/mjt
From: Stephen Hemminger <[email protected]>
Date: Tue, 31 Oct 2006 10:22:22 -0800
> On Tue, 31 Oct 2006 01:31:54 -0800 (PST)
> David Miller <[email protected]> wrote:
>
> > From: Peter Hicks <[email protected]>
> > Date: Tue, 31 Oct 2006 09:25:50 +0000
> >
> > [ Discussion belongs on [email protected], added to CC: ]
> >
> > > I have a dual 3GHz Xeon machine with a 2.4.21 kernel and thousands (15k+) of
> > > ipip tunnel interfaces. These are being used to tunnel traffic from remote
> > > routers, over a private network, and handed off to a third party.
> > ...
> > > Is it possible to speed up creation of the interfaces? Currently it takes
> > > around 24 hours. Is there are more efficient way to handle a very large
> > > number of IP-IP tunnels? Would upgrading to a 2.6 kernel be of use?
> >
>
>
> 2.4 has a several N^2 searches for interfaces (and is in deep freeze by now).
> 2.6 had several changes to handle 1000's of interfaces.
Oops I didn't notice this was with 2.4.x. Indeed, 2.4.x definitely
cannot handle large numbers of networking interfaces at all without
major surgery. 2.6.x should handle this significantly better.