One of the benefits of being the target of trolls and a variety of other
malicious groups is that my servers have been subjected to non-stop denial
of service attacks, hacking, email and spam floods, and just about every
conceivable type of malicious activity someone with a computer is capable
of perpetrating. I get thousands of spam emails a day with not less that
50 spambots every minute of every day lambasting my servers.
And I have to say, it's been a very rewarding experience from a technology
perspective. Since I develop forensics software, this situation has never
ceased to provide me with a wonderful testing environment for my forensics
software. It got so bad, I had to write a firewall that blocks network
traffic a lot like wikipedia does when dealing with malicious bots and
this firewall is interactive with postfix and other apps.
One of the biggest problems of dealing with armies of spambots is even if
you use greylisting or RBL blocking, it does not stop the connection
flooding when they come back over and over and over and over.
iptables is just too cumbersome and memory comsumptive to work well and
has a shitty app inteface so I wrote one with a kernel level database and
combined it with postfix. This firewall actually drops packets on the
floor by port, or in their entirety by IP address to deal with these
jerks.
The code is a kernel module that will build an RBL database to disk and it
will cache up to 500,000 IP addresses efficiently on a 1GB home personal
computer. The more memory you have, the more IP addresses you can cache.
It is configurable and possible to hold millions of them if you have 4GB
of memory in the server.
You will need one of the forenfs patches to properly patch net/core/dev.c
with the ff_filter hook. Included is postfix-2.4.5 which I have modified
and integrated into the ff module. My mail server uses the standard
rejection tests to determine whether or not to block an IP address FOREVER
on port 25.
ftp://ftp.wolfmountaingroup.org/pub/forenfs/patches/ffs-2.6.27.8-el5-12-11-08.patch
ftp://ftp.wolfmountaingroup.org/pub/ff/postfix/postfix-2.4.5.tar.gz
ftp://ftp.wolfmountaingroup.org/pub/ff/ff.tar.gz
It reduces the spam traffic from botnets by 98% on average since any
system identified as a bot get perma-blocked. It's also useful if you
have folks who like to perform DOS attacks or password guessing. The
IOCTL interface can be quickly integrated into just about any app quickly,
giving you the power to be a Wikipedia-style admin with your network
traffic and either block by port or ban, and its a lot faster and more
flexible than the iptables netfilter interface.
This software is designed specifically for active blocking with email
programs and other servers.
Enjoy,
Jeff
On Thu, Jan 08, 2009 at 07:23:43PM -0700, [email protected] wrote:
> iptables is just too cumbersome and memory comsumptive to work well and
> has a shitty app inteface so I wrote one with a kernel level database and
> combined it with postfix. This firewall actually drops packets on the
> floor by port, or in their entirety by IP address to deal with these
> jerks.
>
> The code is a kernel module that will build an RBL database to disk and it
> will cache up to 500,000 IP addresses efficiently on a 1GB home personal
> computer. The more memory you have, the more IP addresses you can cache.
> It is configurable and possible to hold millions of them if you have 4GB
> of memory in the server.
why didn't you use ipset for that ? It's designed exactly for this usage
and is a lot easier to use than plain iptables for dynamic filtering.
Willy
> On Thu, Jan 08, 2009 at 07:23:43PM -0700, [email protected]
> wrote:
>> iptables is just too cumbersome and memory comsumptive to work well and
>> has a shitty app inteface so I wrote one with a kernel level database
>> and
>> combined it with postfix. This firewall actually drops packets on the
>> floor by port, or in their entirety by IP address to deal with these
>> jerks.
>>
>> The code is a kernel module that will build an RBL database to disk and
>> it
>> will cache up to 500,000 IP addresses efficiently on a 1GB home personal
>> computer. The more memory you have, the more IP addresses you can
>> cache.
>> It is configurable and possible to hold millions of them if you have 4GB
>> of memory in the server.
>
> why didn't you use ipset for that ? It's designed exactly for this usage
> and is a lot easier to use than plain iptables for dynamic filtering.
>
> Willy
No database to store the 500,000+ addresses you will harvest in about 2
months, that's why. The one I did uses an lru cached database that runs
in the kernel, and not userspace, so you can filer real time, and manage
the database real time.
Jeff
>
>
On Fri, Jan 09, 2009 at 12:36:06AM -0700, [email protected] wrote:
> > On Thu, Jan 08, 2009 at 07:23:43PM -0700, [email protected]
> > wrote:
> >> iptables is just too cumbersome and memory comsumptive to work well and
> >> has a shitty app inteface so I wrote one with a kernel level database
> >> and
> >> combined it with postfix. This firewall actually drops packets on the
> >> floor by port, or in their entirety by IP address to deal with these
> >> jerks.
> >>
> >> The code is a kernel module that will build an RBL database to disk and
> >> it
> >> will cache up to 500,000 IP addresses efficiently on a 1GB home personal
> >> computer. The more memory you have, the more IP addresses you can
> >> cache.
> >> It is configurable and possible to hold millions of them if you have 4GB
> >> of memory in the server.
> >
> > why didn't you use ipset for that ? It's designed exactly for this usage
> > and is a lot easier to use than plain iptables for dynamic filtering.
> >
> > Willy
>
>
> No database to store the 500,000+ addresses you will harvest in about 2
> months, that's why. The one I did uses an lru cached database that runs
> in the kernel, and not userspace, so you can filer real time, and manage
> the database real time.
ipset runs in kernel too, you just add/remove entries from userspace
without having to touch all other ones. It has no problem storing one
million addresses and doing fast lookups on them.
I'm not dismissing your work, I just think it's a duplicate effort.
Also, since you're speaking about botnets, you should support automatic
expiration of those addresses, because almost all those addresses are
dynamic and will match a bot for a small amount of time, then match a
normal non-infected user. One of the reasons you found 500k addresses
might very well be because each bot appears one hundred times at different
addresses.
Willy
... snip
> ipset runs in kernel too, you just add/remove entries from userspace
> without having to touch all other ones. It has no problem storing one
> million addresses and doing fast lookups on them.
>
> I'm not dismissing your work, I just think it's a duplicate effort.
>
> Also, since you're speaking about botnets, you should support automatic
> expiration of those addresses, because almost all those addresses are
> dynamic and will match a bot for a small amount of time, then match a
> normal non-infected user. One of the reasons you found 500k addresses
> might very well be because each bot appears one hundred times at different
> addresses.
>
> Willy
>
>
You should go and look at the code, 1) the window of addresses cached in
memory is designed to act as an LRU windows for the addresses stored in
the database to use less memory, so no, the in-memory only ip tables is
primitive in comparison 2) the database can just keep growing ad growing
3) the code I posted also loads the database if the system reboots, so
your applications remember all those botnet addresses 4) their is the
ability to set a timer to expire and recycle the oldest addresses (while
still remembering all of them).
>From my experience with dealing with these systems, and observation of how
RBL databases work, when an infected system gets blacklisted, it stays
that way until the user goes to the websites and requests removal. I have
found these zombie systems tend to stay that way, and no, by default you
NEVER want to unblock them for at least 6 months.
Jeff
On Fri, Jan 09, 2009 at 01:15:06AM -0700, [email protected] wrote:
> You should go and look at the code, 1) the window of addresses cached in
> memory is designed to act as an LRU windows for the addresses stored in
> the database to use less memory, so no, the in-memory only ip tables is
> primitive in comparison
I was not speaking about iptables but ipset which is an iptables extension.
>From my memories, you can have addresses stored as bitmaps, where one bit
equals one address. This would lead to less than 100 kB of RAM for your
500k addresses. But I agree that the concept of the LRU cache is interesting.
> 2) the database can just keep growing ad growing
> 3) the code I posted also loads the database if the system reboots, so
> your applications remember all those botnet addresses 4) their is the
> ability to set a timer to expire and recycle the oldest addresses (while
> still remembering all of them).
IIRC, this is also supported in ipset (I just have not looked at it for a
while now).
> From my experience with dealing with these systems, and observation of how
> RBL databases work, when an infected system gets blacklisted, it stays
> that way until the user goes to the websites and requests removal. I have
> found these zombie systems tend to stay that way, and no, by default you
> NEVER want to unblock them for at least 6 months.
This is stupid considering that most of them change their IP address every
24 hours, or at most every 7 days. This is just used to show that spam rate
drops, hiding the fact that valid mails drop for similar reasons. For your
own use, you might consider that you'll never receive mails from people
hosted at the same ISP as the bots you block, but doing this on a large
scale or for companies who do their business around e-mail is plain stupid.
I'm on a static IP, but a lot of people I know are not. It would be
unfair to block them from posting to, say, LKML just because the week
before, someone with their address had been sending spam. And no, it
does not help getting the problem solved since the only users annoyed
are not the ones with the faulty installation.
Willy
... snip
>> From my experience with dealing with these systems, and observation of
>> how
>> RBL databases work, when an infected system gets blacklisted, it stays
>> that way until the user goes to the websites and requests removal. I
>> have
>> found these zombie systems tend to stay that way, and no, by default you
>> NEVER want to unblock them for at least 6 months.
>
> This is stupid considering that most of them change their IP address every
> 24 hours, or at most every 7 days. This is just used to show that spam
> rate
> drops, hiding the fact that valid mails drop for similar reasons. For your
> own use, you might consider that you'll never receive mails from people
> hosted at the same ISP as the bots you block, but doing this on a large
> scale or for companies who do their business around e-mail is plain
> stupid.
>
> I'm on a static IP, but a lot of people I know are not. It would be
> unfair to block them from posting to, say, LKML just because the week
> before, someone with their address had been sending spam. And no, it
> does not help getting the problem solved since the only users annoyed
> are not the ones with the faulty installation.
>
> Willy
>
>
Allowing a server on a dynamic IP range to act as a mail server is what is
stupid. Most email blockers also block dynamic IP ranges since these
systems are typically the ones most commonly infected, and the RBL
databases. They just use a different technique, they base it on DNS maps
of internet address ranges, and not behavior. It will only block the smtp
port for these systems -- and mail servers originating from dynamic ranges
should be blocked anyway -- and are by most commercial email gateway
boxes.
Jeff
[email protected] wrote:
> Allowing a server on a dynamic IP range to act as a mail server is what is
> stupid.
Sorry to be argue semantics, but I think it's SMTP clients that are the
major cause of spam. Many, probably most, ISPs provide an SMTP relay
for their subscribers, and blocking these dynamic addresses doesn't
present a hard impediment for delivery of their mail to you. None the
less, to say that people on a dynamic IP address should be *obliged* to
go through a relay is offensive. (Not that you did say that, but I
think it's what you had in mind.)
> [email protected] wrote:
>> Allowing a server on a dynamic IP range to act as a mail server is what
>> is
>> stupid.
>
> Sorry to be argue semantics, but I think it's SMTP clients that are the
> major cause of spam. Many, probably most, ISPs provide an SMTP relay
> for their subscribers, and blocking these dynamic addresses doesn't
> present a hard impediment for delivery of their mail to you. None the
> less, to say that people on a dynamic IP address should be *obliged* to
> go through a relay is offensive. (Not that you did say that, but I
> think it's what you had in mind.)
>
That's good point, but not what I see in actual practice. These zombie
systems don't seem to use the smtp gateway for their ranges, they send the
emails direct. From my observations, most of these systems tend to stay
zombies, since almost all of them are windows systems.
I have discovered that by using this method to block them, once they open
a connection and I start dropping traffic while the connection is left
open it "tarpits" the bots and causes a cascade failure in the botnet
network for about 20 minutes until the systems start skipping the mail
server doing the bocking.
Jeff
On Fri, 09 Jan 2009, Willy Tarreau wrote:
> why didn't you use ipset for that ? It's designed exactly for this usage
> and is a lot easier to use than plain iptables for dynamic filtering.
Any ideas when it will hit mainline? ipsets and PF_RING are the only ways
to get two important jobs done: non-trivial firewalling on high-speed
links, and packet capture in said links... and neither is in mainline AFAIK.
--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh
On Fri, Jan 09, 2009 at 10:40:31PM -0200, Henrique de Moraes Holschuh wrote:
> On Fri, 09 Jan 2009, Willy Tarreau wrote:
> > why didn't you use ipset for that ? It's designed exactly for this usage
> > and is a lot easier to use than plain iptables for dynamic filtering.
>
> Any ideas when it will hit mainline?
no idea.
> ipsets and PF_RING are the only ways
> to get two important jobs done: non-trivial firewalling on high-speed
> links, and packet capture in said links...
yes, that's a bit right.
> and neither is in mainline AFAIK.
Well, you should ask Joszef for the former and Luca for the later :-)
Regards,
Willy